Day 19 of 2160 minCipher CadetHands-on lab
Day 19: Unsafe Output Handling and Fake Packages
Taught by Mr Elite Updated
By the end of today you can
- Show how model output rendered as HTML or Markdown can run attacker content in a browser
- Explain why model output must be treated like any untrusted input
- Spot a hallucinated package name and check it before installing
- Write the safe handling rule for each kind of output
Warm-up
Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.
In Day 18, why do allow lists and human approval still work when the model is fooled?
They are enforced outside the model, in ordinary code or by a person, so the model's decision cannot override them.
What does least privilege mean for an agent's tools?
The agent only gets the tools the task needs, so a dangerous tool is simply not available to be hijacked.
For the last four days we worried about what goes into a model. Today we flip it. A chatbot replies, the app takes that reply and puts it straight on the page, or straight into a terminal command, and nobody asks whether the reply is safe. The model did not need to be hacked in any dramatic way. The app simply trusted its words.
The second half of today is a cousin of the same problem. A coding assistant confidently suggests a package that does not exist, and a developer runs the install. If someone has registered that invented name with bad code inside, the assistant just delivered the attack. You will practise both today, on your own machine and in the labs.
Model output is untrusted input
On Day 18 we saw that model output can drive actions. Today's rule follows from that: anything a model writes should be treated exactly like text typed by a stranger on the internet. Web developers learned this long ago for form fields. You check it, escape it, and never run it as code.
Why is the model untrusted? Because you can steer it. Through direct injection, a poisoned document or a poisoned knowledge base, an attacker can influence what it writes. So whatever the app does next with that text, the attacker has a say in it.
- Unsafe output handling
- An app uses model output in a risky place (a web page, a shell, a database query) without checking or escaping it. This is OWASP LLM05 on the Top 10 you mapped on Day 3.
- XSS (cross-site scripting)
- An attack where attacker-controlled content runs as code inside another user's browser session.
- Escaping
- Converting special characters so they display as plain text instead of being treated as code.
Quick check
A chatbot reply is inserted into a web page. Which view is safest?
Prompts can be bypassed. Escaping in ordinary code does not depend on the model behaving.
How Markdown and HTML rendering becomes an attack
Many chat apps turn Markdown into HTML so replies look tidy. That conversion is a rendering step, and rendering is where text becomes something the browser acts on. If the app inserts the result into the page without sanitising it, the reply can carry active content.
There is a quieter version too. A Markdown image looks like . The browser fetches that address automatically, with no click. If the address contains conversation data, loading the picture sends that data to whoever runs the server. Nobody sees a script, only a broken or tiny image. This is why a filter that only blocks raw HTML can give false comfort.
Watch me do it: the same reply, rendered two ways
We will build a tiny page on your own machine. It pretends a model returned a hostile reply, then shows it once the unsafe way and once the safe way. Nothing leaves your computer, and the payload only opens a harmless pop-up.
<button onclick="unsafe()">Unsafe</button>
<button onclick="safe()">Safe</button>
<div id="box"></div>
<script>
const reply = '<img src=x onerror="alert('output was trusted')">';
function unsafe(){ box.innerHTML = reply; }
function safe(){ box.textContent = reply; }
</script>Look at what happened. The model never needed to run anything. The page did it, because the developer chose innerHTML. In real apps the fix is usually a sanitiser library with a strict allow list of tags, plus a content security policy that limits where images and scripts can load from.
Package hallucination and slopsquatting
A hallucination is when a model states something false with total confidence. Coding assistants do this with package names. They suggest a library that sounds right, like a plausible mix of real names, but that nobody ever published. Worse, models tend to repeat the same invented name across many users.
Slopsquatting is the attack built on that. An attacker collects names that assistants keep inventing, registers them on a public registry such as PyPI or npm, and puts malicious code inside. A developer who pastes the suggested install command now runs the attacker's code. The model's output drove an action, which is today's theme again.
Your defence is a short habit: verify before you install. Ask whether the package exists, how old it is, who maintains it, and whether it links to a real source repository. A brand new name with no history that an AI just recommended deserves suspicion.
curl -s -o /dev/null -w "%{http_code}n" https://pypi.org/pypi/requests/json
curl -s -o /dev/null -w "%{http_code}n" https://pypi.org/pypi/zz-not-a-real-pkg-83712/jsonQuick check
Why is a 200 response alone not proof that a suggested package is safe?
Existence only shows the name is taken. An attacker who got there first makes it exist too.
Your labs today
Both labs run in the SecurityElites practice environment, so everything you try stays inside it. The first, ai-markdown-rendering-injection-1, is the rendering problem you just saw. The second, ai-package-hallucination-1, is the slopsquatting problem. Do the Markdown lab first, then the package lab, and keep short notes on each for the mission.
Your turn
AI Markdown-Rendering Injection
DocBot renders its replies from Markdown to HTML and blocks raw HTML, but allows Markdown images because they are not executable. Today's idea is that a filter can block the obvious route and still leave another one open, so this is the right place to test it.
What to try
What to notice
- Which output feature the developer assumed was harmless, and why that assumption was wrong
- That the model is not the broken part. The way the app handles its reply is
- Whether the assistant gives the same invented name when you ask in different ways
- How a verified, established package differs from a name with no history
If it did not work
Relying on the system prompt to stop the model writing dangerous output
Fix it in code. Escape or sanitise the reply at the point it is displayed or used, so it holds even when the model is fooled.
Blocking script tags and assuming rendering is safe
List every feature the renderer still allows, including images and links, and test each. Use a sanitiser with an allow list plus a content security policy.
Treating a package as safe because it exists on the registry
Check age, maintainers, download history and source repository. Prefer names you can verify from official documentation.
Running an AI-suggested install command without reading it
Read the package name first, look it up, and install inside a throwaway environment if you are unsure.
Recap
- Model output is untrusted input and needs the same care as anything a stranger types
- Rendering is where text becomes action, so unsafe HTML or Markdown handling can lead to XSS
- Image links can carry data out because the browser loads them automatically
- Escape or sanitise in code, and add a content security policy as a second layer
- Assistants invent package names, and attackers register them in slopsquatting
- Verify a package's existence, age, maintainers and source before installing
Cheat sheet
textContent vs innerHTML- textContent shows text safely. innerHTML parses it as markup and is risky with untrusted output.
Sanitiser with allow list- Strips everything except tags and attributes you have explicitly approved.
Content security policy- Browser rule limiting where scripts and images may load from.
curl -s -o /dev/null -w "%{http_code}" URL- Prints only the HTTP status code of a request.
Slopsquatting- Registering a name AI tools tend to invent, then filling it with malicious code.
OWASP LLM05- Improper output handling.
Before the next day
Mission: Two labs, two safe-handling sentences
Finish both labs and write one clear sentence per lab on the safe way to handle the output. Writing the fix in a single sentence trains you to state a remediation plainly, which you will need in Day 20 and in your capstone report.
About 40 min
You are done when: day19-note.md with two sentences (one per lab), two screenshots, and the updated cheat sheet.
Want more? Extend the output-demo.html page with a small allow-list sanitiser, or write a three-line pre-install checklist for AI-suggested packages.
Quiz
4 questions. Your score is saved with your progress.
1. An app inserts a chatbot reply into a page using innerHTML. What is the core mistake?
Model output can be steered by an attacker, so inserting it as markup lets attacker content run.
2. A filter blocks raw HTML but allows Markdown images. Why can that still be exploited?
An allowed feature can still be a channel. The image request itself sends the data.
3. What makes slopsquatting practical for an attacker?
The consistency of the hallucinated name lets an attacker predict what to register.
4. Which control best holds when the model is fooled?
Controls enforced outside the model do not depend on the model behaving.
Finish the day
Mark the day complete to save your progress.
Next: Day 20 pulls everything together. You will map defences to every OWASP LLM Top 10 item and learn to structure a finding for a report, using the notes and cheat sheet you have been building.
Your notes
Private to this device. Write what you want to remember from today.