Day 9 of 960 minCipher CadetHands-on lab
Day 9: Capstone: Threat Model, Test and Report
Taught by Mr Elite Updated
By the end of today you can
- Draw an architecture diagram and mark trust boundaries for a small AI assistant
- Write a threat model with one threat per boundary
- Review a RAG ingestion plan in the lab and record what you found
- Write a short report with three ranked fixes and three next learning goals
Warm-up
Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.
Why do you record a baseline refusal before testing a bypass?
It proves the bot refused the plain request, so a later success shows your bypass changed the outcome.
How do you rate the severity of a finding?
By impact and ease, then explain the rating in one sentence.
You have spent eight days learning pieces: tokens, system prompts, tools, boundaries, test plans. Today you put them together the way a real engagement ends, with a document someone else can read and act on.
Picture a team about to connect company documents to an AI assistant. They ask you one question: is this safe to ship? A pile of notes will not answer it. A short report with a diagram, a list of threats and three ranked fixes will. That is what you build today.
Choose your target and map it
First we need something to assess. Today our target is a small fictional assistant: a company helpdesk bot that answers staff questions from internal documents using retrieval (RAG, short for retrieval-augmented generation, where the app fetches relevant document chunks and pastes them into the prompt). It matches today's lab, so your report and your lab work join up.
Everything here is paper and practice labs. You will not test any real system. A threat model is a structured way of asking what could go wrong before it does, and a diagram is where it always starts, because you cannot protect parts you have not named.
- Ingestion
- The step where documents are collected, split and stored in the index the assistant searches.
- Trust boundary
- A line where data passes between parts that deserve different levels of trust.
- Finding
- One problem you can describe, show evidence for, rate and fix.
- Threat model
- A written list of what could go wrong in a system, tied to the places in the design where it could happen.
Now the architecture. For the helpdesk bot I use six parts: the staff user, the web chat front end, the application server, the model API, the document index, and the document sources (shared drive, wiki). Draw them as boxes with arrows showing the flow of one question, then the flow of documents into the index.
[Staff user] -> [Chat web page] -> [App server] -> [Model API]
|
v
[Document index] <- [Shared drive, wiki]Mark trust boundaries and list threats
Earlier in the course we learned that trouble lives where data crosses a line. On your diagram, draw a dashed line wherever data moves between parties with different trust. For our bot there are four: user to web page, app server to model, document sources into the index, and index content into the prompt.
Why these four? The user is anyone who can type, so their words are untrusted. The model API is run by someone else, so what you send it leaves your control. Documents come from many authors, so they are untrusted too. And retrieved text lands in the prompt right next to your own rules, which is the most surprising crossing of all.
For each boundary, ask one question: what could go wrong here? A simple method is to ask about three things: can someone say what they should not (spoofing or injection), can someone see what they should not (disclosure), and can something be made to do what it should not (misuse). Write one threat per boundary, in one sentence.
B1 User -> chat page: user types an injection to override the rules
B2 App server -> model: system prompt leaks in the reply
B3 Sources -> index: someone plants instructions in a document
B4 Index -> prompt: the bot shows a document the user may not readQuick check
A staff member edits a wiki page to include hidden instructions for the bot. Which boundary is crossed first?
The planted text enters through the document source on its way into the index, before any user asks a question.
Watch me do it: draft the report skeleton
Before the lab, I set up the file that holds everything. A report is easier to fill in than to write from blank. Follow along on your own machine, using only your own files.
DAY 9 REPORT: Helpdesk RAG assistant (fictional)
1. Architecture diagram
2. Trust boundaries
3. Threat model (one threat per boundary)
4. Test results (Day 8 table + today's lab)
5. Ranked fixes (1 to 3)
6. Next learning goals (3)What just happened: you turned eight days of scattered notes into a structure with a slot for every claim. Anything you cannot place under a heading is probably not yet a finding. Anything with a heading but no content is a gap you now know about.
Lab: review a RAG ingestion plan
Now we test the design. The lab called What Goes in the Index? gives you a company ingestion plan with seven decisions. Three of them let the assistant repeat planted content or show people documents they should not see. Your job is to find which ones and say why, which is exactly a design review, the cheapest and best moment to catch these problems.
Quick check
Why is reviewing an ingestion plan useful before any chatbot exists?
Content that never enters the index cannot be repeated or leaked, so design fixes remove the problem early.
Write findings and rank fixes
A good finding has five parts: what, where, evidence, severity, fix. Keep each to a line or two. Here is the shape using a made-up example. Notice that the evidence line names something a reader could go and look at.
Finding: Wiki pages are indexed without review
Where: Boundary B3 (sources to index)
Evidence: Lab plan, decision about auto-importing all pages
Severity: High. Easy to do, and the bot repeats the text to everyone
Fix: Index only approved folders and record who added each itemRanking is a decision, and being willing to make it is what makes a report useful. A team with a week to spare will do your first fix and probably your second. Make sure those two matter.
Review the course and choose what next
Look back over nine days. You learned how text becomes tokens, how a context window and system prompt shape answers, how applications and agents connect to tools, where trust boundaries sit, and how to plan and run tests safely. Today you joined those into one deliverable.
Add section 6. Pick three goals from these routes: agents and tool abuse, RAG security in more depth, defensive labs where you fix what you broke. Write one sentence on why you chose each and one small first step, such as a lab to complete this week.
6. Next learning goals
- Agents and tool abuse: I want to see how tool permissions fail. First step: finish one agent lab this week.
- RAG security in depth: B3 and B4 gave me my best findings. First step: re-read my flagged decisions.
- Defensive labs: I want to practise fixing. First step: pick one finding and write its fix as a task.Your turn
What Goes in the Index?
This lab puts the design-review half of red teaming into practice. Instead of attacking a bot, you read a plan and judge it against the boundaries you drew, which is where many real RAG problems are cheapest to prevent.
What to try
What to notice
- Which of your four boundaries each risky decision touches
- How a harmless-looking convenience choice becomes a risk
- Whether your three flagged items share a common cause
If it did not work
Writing a report that lists attacks but no evidence
For every finding, point to the exact lab decision or recorded prompt and result that supports it.
Marking boundaries on the diagram but not tying threats to them
Number each boundary and label each threat with the same number.
Giving every finding the same severity
Rate by impact and ease, then add a one-sentence reason so the ranking can be challenged.
Leaving fixes vague, such as 'make it more secure'
State a specific change someone could put on a task list, for example 'index only approved folders'.
Recap
- A threat model starts with a diagram and trust boundaries, then one question per boundary
- Design reviews catch RAG problems before any attack is needed
- Content that enters the index can be repeated or exposed to the wrong people
- A finding needs what, where, evidence, severity and fix
- Ranking three fixes is more useful than listing many
- Learning continues with agents, RAG depth, tool abuse and defensive labs
Cheat sheet
Trust boundary- Where data crosses between parts with different trust
Ingestion- Collecting and storing documents in the searchable index
Finding format- What, where, evidence, severity, fix
Severity- Impact and ease, with a one-sentence reason
Report order- Diagram, boundaries, threats, results, fixes, next goals
Before the next day
Mission: Finish your capstone report
Pull everything into one file called day9-report.txt so that a reader who was not on this course can understand what you built, tested and recommend. This is the proof of your nine days of work.
About 45 min
You are done when: A file day9-report.txt containing diagram, boundaries, threat model, results, three ranked fixes and three learning goals.
Want more? Add a short paragraph on what you could not test and what you would test next given more time.
Quiz
4 questions. Your score is saved with your progress.
1. Why do you mark trust boundaries before listing threats?
Boundaries show you where to look, so each threat has a clear place in the system.
2. A planted instruction in an indexed document is dangerous mainly because:
Retrieved text enters the prompt, and the model cannot reliably tell data from instructions.
3. Which fix best belongs at the top of a ranked list?
Ranking by risk reduction against effort gives the team the best first action.
4. What makes a finding believable to a reader?
Evidence lets someone else repeat or verify the result, which is what turns a claim into a finding.
Finish the day
Mark the day complete to save your progress.
What next: There is no Day 10, so your next step is your own. Pick one of your three goals, such as agents and tool abuse, deeper RAG security or defensive labs, and start it this week while today's report is fresh.
Your notes
Private to this device. Write what you want to remember from today.