Day 3 of 2160 minCipher Cadet
Day 3: Threat thinking and the OWASP LLM Top 10
Taught by Mr Elite Updated
By the end of today you can
- Describe any AI feature as an attacker, an asset and an impact
- Name all ten OWASP LLM Top 10 items and explain each in one plain sentence
- Tag the entry points from your Day 2 diagram with the Top 10 item they relate to
- Say which items we will practise on which days of this course
Warm-up
Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.
Name three kinds of text that share the context window with your message.
Any three of: the system prompt, conversation history, retrieved documents, tool results.
Why is the developer's system prompt the only truly trusted input in the diagram?
The developer wrote it and controls it. Everything else can be influenced by users or outside content.
Yesterday you drew a map of a support bot and marked every place text can get in. That map is useful, but a map alone does not tell you where to look first. Ten entry points do not all carry the same risk. A customer typing a rude message is not the same as a poisoned help article that every customer's bot reads.
Today I give you two tools to sort that out. The first is a three-part question that turns any entry point into a clear risk. The second is the OWASP LLM Top 10, a shared list of the ten most common ways AI applications go wrong. By the end you will speak the same language as every AI security report you will ever read.
Start from your Day 2 map
Open your day02 folder and look at your diagram and your numbered list of entry points. Each arrow you marked is a place where text reaches the context window. Remember that the model reads one stream, so any of those arrows could carry an instruction the developer never wrote.
Security people do not try to attack everything at once. They ask what matters, who might go after it, and what happens if they succeed. That habit is called threat thinking. It stops you wasting a day on a harmless quirk while missing the flaw that leaks customer data.
Here is a quick comparison. Entry point A is the chat box, and the attacker is any customer, so it is very easy to reach. Entry point B is a help article that only staff can edit, so it is harder to reach. But if B is poisoned, every customer's conversation is affected at once. Easy to reach and wide in effect are different measures, and we need both.
- Threat model
- A short, structured description of what could go wrong with a system and who could make it happen.
- OWASP
- The Open Worldwide Application Security Project, a non-profit that publishes free security guidance used across the industry.
- Entry point
- Any place where outside text or data can reach the model. You listed yours yesterday.
Quick check
Why do we rank entry points instead of testing them all equally?
Entry points differ in who controls them and what they can reach, so we spend time where the risk is highest.
Attacker, asset, impact
A simple threat model needs only three answers. Who is the attacker, meaning who can put text into this entry point? What is the asset, meaning the thing worth protecting? What is the impact, meaning what bad outcome follows if the attacker reaches the asset?
For the support bot, an attacker might be an ordinary customer, or a stranger who wrote a web page the bot reads. Assets might be the hidden system prompt, other customers' order details, or the bot's ability to issue refunds. Impact might be leaked data, money lost, or a bot that embarrasses the company in public.
Write the three answers as one sentence: an attacker who can do X to entry point Y could reach asset Z and cause impact W. If you cannot fill in every part, you do not yet understand the risk, and that tells you what to investigate next.
Let me show the difference a good sentence makes. Weak: the chat box can be abused. Strong: a customer who types instructions into the chat box could make the bot reveal its hidden prompt, exposing discount rules the company wants private. The strong version names the attacker, the entry point, the asset and the harm, so a manager can decide whether to care.
Also be honest about who the attacker really is. Some attackers are the users themselves. Others never touch the bot at all, such as someone who publishes a web page the bot later reads. When you struggle to name an attacker, ask who can write to that entry point, directly or by tricking someone who can.
Quick check
Which sentence is a complete threat model statement?
Only the second names an attacker, an entry point, an asset and an impact.
A guided tour of the OWASP LLM Top 10
OWASP keeps a ranked list of the ten biggest risks for applications built on large language models. The numbering below follows the 2025 edition. Wording and order change between editions, so check owasp.org for the current version once you finish the course. Here is each item in one plain sentence.
- LLM01 Prompt Injection
- Text an attacker supplies, directly or hidden in content, changes what the model does.
- LLM02 Sensitive Information Disclosure
- The model reveals private data such as personal details, secrets or internal records.
- LLM03 Supply Chain
- A model, library, dataset or plugin you depend on is already compromised or unsafe.
- LLM04 Data and Model Poisoning
- Bad data planted in training or tuning material quietly bends the model's behaviour.
- LLM05 Improper Output Handling
- The app trusts model output and passes it on unchecked, so that output becomes an attack.
- LLM06 Excessive Agency
- The model is given more tools, permissions or freedom than its job needs.
- LLM07 System Prompt Leakage
- Hidden instructions, and anything sensitive inside them, are exposed to users.
- LLM08 Vector and Embedding Weaknesses
- Flaws in how retrieved knowledge is stored and fetched let attackers poison or read it.
- LLM09 Misinformation
- The model states false things confidently, and people or systems act on them.
- LLM10 Unbounded Consumption
- Uncontrolled use of the model runs up cost, exhausts capacity or lets attackers copy it.
Notice that several of these are not about clever prompts at all. LLM03, LLM05 and LLM06 are ordinary software security problems wearing an AI costume. Good red teamers test the whole application, not just the chat box.
It helps to group them. LLM01 and LLM07 concern the instructions going in. LLM02, LLM04 and LLM08 concern data, either leaking out or being poisoned on the way in. LLM03, LLM05, LLM06 and LLM10 concern the software and permissions around the model. LLM09 concerns the people and systems that believe what the model says.
One more point before you try it. A single attack often touches several items. An attacker who plants instructions in a help article is using LLM01, the article store is LLM08, and if the bot then issues a refund it did not need permission for, that is LLM06. Tagging several items to one entry point is normal.
Quick check
A coding assistant is given permission to delete files and send emails, although it only needs to read code. Which item is this?
Giving a model more tools and permissions than its task needs is the definition of excessive agency.
Watch me do it: threat model one entry point
Now I will take one entry point from the support bot and turn it into a tagged risk. Do this alongside me on your own machine. Nothing here touches any real system, it is all notes.
mkdir day03
cd day03
touch threat-notes.md
lsmkdir day03
cd day03
New-Item threat-notes.md
dirEntry point:
Attacker:
Asset:
Impact:
Sentence:
Top 10 tags:Entry point: help article the bot retrieves
Attacker: anyone who can edit or submit a help article
Asset: the bot's refund tool and customer order data
Impact: bot follows hidden orders and leaks or misuses data
Sentence: An attacker who edits a help article could steer the bot into misusing refunds, harming the company's money and trust.
Top 10 tags: LLM01, LLM08, LLM04Look at what we just produced. In six lines, a vague worry about help articles became a specific risk with a named attacker, a named asset and a harm. It also carries three tags that tell a reader which parts of the Top 10 apply. That is the whole method, and you will repeat it for every entry point you find.
Your turn: tag your whole map
This is today's practice, and it runs entirely on your own machine with your own notes. There is no lab today, because the skill is thinking, not typing attacks.
Priority order: 4, 1, 3, 2
Why: 4 reaches refunds and every customer reads it; 1 is easy to reach but low impactWhere we practise each item
You will not meet all ten in the same depth. LLM01 runs through most of the course, because injection is how attackers reach everything else. Here is the plan so you know why each day exists.
- Days 4 to 9
- LLM01 and LLM07: direct injection, system prompt leaks, jailbreaks and filter bypasses.
- Days 10 to 13
- LLM01 indirect injection, plus LLM08 and LLM04 through knowledge base poisoning.
- Days 14 to 18
- LLM06 Excessive Agency and LLM01 against agents, tools and memory, with LLM02 data leaving through tools.
- Day 19
- LLM05 Improper Output Handling, LLM09 Misinformation and the LLM03 supply chain angle of fake packages.
- Day 20
- Defences for all ten, including LLM10, and how to write findings.
When you reach each day, look back at your threat-notes.md file. The tags you wrote today tell you which of your entry points that day's technique will test. That link between your own map and the daily practice is what makes this course a method and not a list of tricks.
Quick check
On which kind of day will you mostly practise LLM01?
Injection appears in direct, indirect, agent and tool attacks, so it threads through most of the course.
If it did not work
Memorising names without meaning
For every item, write a one-sentence explanation in your own words. If you cannot, you only know the label.
Skipping the impact line in the threat model
Always finish the sentence with the harm. A risk with no impact is not yet a risk.
Treating the Top 10 as a checklist that is complete
It is a starting list of common problems. Real systems can fail in ways no list names, so keep your Day 2 map open.
Confusing LLM01 and LLM07
LLM01 is about changing the model's behaviour. LLM07 is about exposing the hidden instructions. One attack can cause both.
Recap
- Threat thinking asks who the attacker is, what the asset is and what the impact would be
- Impact is what makes a finding matter to the people reading your report
- The OWASP LLM Top 10 is a shared ranked list of ten common AI application risks
- Several items are ordinary software problems, so test the whole application
- LLM01 Prompt Injection runs through most of this course
- Your Day 2 entry points now carry risk tags and a priority order
Cheat sheet
Threat model sentence- An attacker who can do X to entry point Y could reach asset Z and cause impact W.
LLM01 Prompt Injection- Attacker text changes what the model does.
LLM02 Sensitive Information Disclosure- Private data is revealed.
LLM03 Supply Chain- A dependency you trust is compromised.
LLM04 Data and Model Poisoning- Bad data bends the model.
LLM05 Improper Output Handling- Output is trusted and passed on unchecked.
LLM06 Excessive Agency- Too many tools or permissions.
LLM07 System Prompt Leakage- Hidden instructions exposed.
LLM08 Vector and Embedding Weaknesses- Retrieval store can be poisoned or read.
LLM09 Misinformation- Confident false output is acted on.
LLM10 Unbounded Consumption- Uncontrolled use causes cost or outage.
Before the next day
Mission: Ten-line Top 10 cheat sheet
Write a cheat sheet with one line per Top 10 item. Each line gives a one-sentence explanation in your own words and a short real-world style example that you invent yourself. Inventing the example forces you to understand the item, and you will reuse this sheet when you write reports on Day 20.
About 30 min
You are done when: A top10-cheatsheet.md file with ten lines, each holding an explanation and an invented example, plus your threat-notes.md file.
Want more? Add a second invented example for LLM01 and LLM06 showing how the two items combine in one attack story.
Quiz
4 questions. Your score is saved with your progress.
1. A user pastes a hidden command into a web page, and your summariser bot obeys it. Which pairing is best?
Attacker-supplied content changing the model's behaviour is prompt injection, here through content the bot reads.
2. Which statement best describes the impact part of a threat model?
Impact is the outcome, such as leaked data or lost money, and it is why a risk is worth fixing.
3. An app renders model output directly as a web page without checking it. Which item applies most directly?
Trusting and passing on model output unchecked is the core of improper output handling.
4. Why is it a mistake to treat the Top 10 as a complete list of everything that can go wrong?
The list guides attention to common problems but cannot replace mapping and thinking about your own system.
Finish the day
Mark the day complete to save your progress.
Next: Tomorrow we start attacking. You will use direct prompt injection to pull a hidden system prompt out of a practice bot, which is LLM01 and LLM07 together, and you will already know which asset you are after.
Your notes
Private to this device. Write what you want to remember from today.