Day 9 of 1260 minCipher CadetHands-on lab
Day 9: Improper Output Handling: Never Trust the Model's Answer
Taught by Mr Elite Updated
By the end of today you can
- Explain why a model's answer must be treated as untrusted input
- Name the four common sinks: web page, database, shell and other tools
- Show safely in a test page how a browser can run code hidden in output
- Choose escaping, validation or an allowlist for each unsafe sink
Warm-up
Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.
Name the three parts of a good finding from yesterday.
What is wrong, why it matters, and how to fix it.
Should an agent remember instructions it found inside untrusted content? Why?
No. Those instructions could be an attacker's, and remembering them would let them steer the agent later.
Picture a support chatbot. A customer asks a question, the model writes a reply, and the app drops that reply straight into the web page. Nobody wrote any attack code. But the customer's message told the model to include a small piece of HTML, the model obliged, and now every agent who opens that ticket runs it in their browser.
The model did what models do. The mistake was the app trusting what came out. Today we follow the answer on its trip through the app and mark every place where that trust is a problem.
The core idea: the answer is untrusted input
Over the last eight days we worried about what goes into a model. Today we flip it and look at what comes out. Here is the rule I want you to carry: the output of a model is untrusted input to whatever system receives it next.
Why untrusted? Because the model's answer is shaped by its prompt, and the prompt can include text from users, web pages, documents and emails. If an attacker can influence any of that, they can influence the answer. The answer is only as trustworthy as the least trustworthy thing the model read.
The OWASP name for getting this wrong is Improper Output Handling. It means the application passes the model's output to another component without checking or cleaning it first.
Let me draw the flow on the whiteboard in plain text. Every arrow after the model is a place where the app has to decide whether to trust what it was handed.
customer message --> prompt --> MODEL --> reply --> [ page | database | shell | tool ]Notice that the customer never touches the destinations directly. The model sits in the middle and passes their influence along. That is why this bug is easy to miss: the app's own code looks like it is doing all the work.
- Sink
- Any place the model's answer ends up and gets used: a web page, a database query, a shell command, another tool.
- Escaping
- Converting special characters so the receiving system treats them as plain text, not as instructions.
- Validation
- Checking that output has the expected shape, such as a number between 1 and 5, and rejecting anything else.
- Allowlist
- A short list of permitted values or actions. Anything not on the list is refused.
Quick check
A model's reply is built from a prompt that included a customer's message. How should the app treat the reply?
The customer's text shaped the reply, so the reply inherits that lack of trust.
The four common sinks
A sink is where the answer lands. Each sink has its own way of going wrong, because each one has its own language it understands.
- Web page sink
- The answer is inserted into HTML. If it contains a script or an event handler, the browser may run it. This is cross-site scripting, often shortened to XSS.
- Database sink
- The answer is placed into a query. Special characters can change what the query does. This is SQL injection.
- Shell sink
- The answer becomes part of a command line. Characters like a semicolon can add extra commands.
- Tool sink
- The answer is passed to another tool or API as its input, such as a refund function or an email sender.
Here is what the risky pattern looks like in code. I am showing these lines so you can recognise them in a review. Do not run them. In each one, the model's text is glued straight into something that has its own command language.
page = "<p>" + reply + "</p>"
query = "SELECT * FROM tickets WHERE dept = '" + reply + "'"
cmd = "ping -c 1 " + replyNotice the pattern. In every case, data crosses into a system that has a command language, and the data is allowed to speak that language. Plain text sinks, like a log line shown as text, are much safer.
Watch me do it: see a browser run output
I will show you why a browser rendering output is dangerous, using a harmless page on your own machine. We will use a pretend model reply and a harmless alert box. Nothing here touches any other system, and no internet connection is needed.
<div id="out"></div>
<script>
var reply = '<img src=x onerror="alert('output ran as code')">';
document.getElementById('out').innerHTML = reply;
</script>Let me explain what just happened. The string looks like an image tag pointing at a file called x, which does not exist. The browser tries to load it, fails, and runs whatever the onerror attribute says. The attacker never wrote a script tag. A plain tag with an event handler was enough.
Now let me fix it. Open the file again and change the word innerHTML to textContent, so the line matches the block below. Save, then refresh the browser tab.
document.getElementById('out').textContent = reply;After refreshing you should see the raw text of the tag on the page, starting with the less-than sign and the letters img, and no popup. Same reply, different sink behaviour. That one word is the difference between safe and unsafe.
Defences: escaping, validation and allowlists
There are three tools, and each fits a different job. Pick the one that matches the sink.
Let me prove the database point on your own machine. You need Python 3, which includes the sqlite3 module. Create a file called sink.py and paste the code below. It uses a throwaway database that lives only in memory, so nothing is saved anywhere.
import sqlite3
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE tickets (id INTEGER, dept TEXT)")
db.execute("INSERT INTO tickets VALUES (1, 'billing'), (2, 'technical')")
reply = "billing' OR '1'='1"
unsafe = "SELECT * FROM tickets WHERE dept = '" + reply + "'"
print(db.execute(unsafe).fetchall())
print(db.execute("SELECT * FROM tickets WHERE dept = ?", (reply,)).fetchall())Run it with python3 sink.py (or python sink.py on Windows). You should see two lines: [(1, 'billing'), (2, 'technical')] and then []. The first query returned every row because the reply changed the query's meaning. The second returned nothing, because the database treated the whole reply as one odd department name.
One more, for the shell and tool sinks. This snippet never runs anything. It only shows the shape of an allowlist, where the model picks a key and your code owns the command.
ACTIONS = {"restart_web": ["systemctl", "restart", "nginx"]}
choice = "restart_web"
if choice in ACTIONS:
print("would run:", ACTIONS[choice])
else:
print("refused:", choice)Quick check
A model picks which department handles a ticket. Which defence fits best?
The valid answers are a small known set, so an allowlist is simple and strong.
Lab: Where Does the Answer Go?
Time to apply this. The lab gives you a support app that uses the model's reply in six places. Your job is to decide which sinks are safe and which are not. Log in to your free SecurityElites account and open the lab named in the lab section below.
Work the way I did in the demos. For each place, ask what receives the reply and whether that thing has a command language. Write your judgement down before you check the feedback, so you learn from the ones you get wrong.
Your turn
Where Does the Answer Go?
This lab shows one app with six places where a model's reply is used. It is the best way to practise tracing an answer and judging each sink, because you see the whole picture instead of one example at a time.
What to try
What to notice
- Plain text sinks, like a log shown as text, are low risk
- The unsafe sinks all hand the answer to something with its own command language
- Several sinks look harmless until you ask what reads the data next
If it did not work
Assuming our own model is a trusted source
Ask what the model read to produce the answer. If any of it came from users or outside content, treat the reply as untrusted.
Filtering the prompt and ignoring the output
Input filters can be bypassed. Always add checks at the sink, where the damage would happen.
Using one generic cleaner for every sink
Escape for the specific destination. HTML, SQL and shell each need different handling.
Asking the model to 'please be safe' as the only control
Instructions to the model are not enforcement. Use code that escapes, validates and allowlists.
Recap
- Model output is untrusted input to the next system
- A sink is anywhere the answer is used: pages, databases, shells, tools
- A browser given output as HTML may run attacker code
- Escape for the destination, validate the shape, allowlist the actions
- Never build shell commands or queries from raw model text
- Structured output is easier to check than free prose
Cheat sheet
Sink- Where the model's answer is used next
XSS- Output run as code in a browser
textContent- Shows a string as plain text, runs nothing
innerHTML- Treats a string as HTML, risky with untrusted text
Parameterised query- Keeps data separate from SQL commands
Allowlist- Only permitted values pass, the rest are refused
Before the next day
Mission: Map the output sinks
Finish the lab, then turn what you learned into a diagram of one application. Drawing the route of the answer is how reviewers find these bugs in real projects, and it prepares you for the capstone audit.
About 30 min
You are done when: A diagram file day9-sinks.png showing every sink for one application, with unsafe ones marked and a fix noted for each.
Want more? Write one what, why, fix finding for your riskiest sink, in the format from Day 8.
Quiz
4 questions. Your score is saved with your progress.
1. Why is a model's reply untrusted even when you built the app yourself?
Users, documents and web pages feed the prompt, so an attacker can steer what the model writes.
2. In the demo, what changed between the unsafe and safe page?
Switching from innerHTML to textContent made the browser display the string rather than run it.
3. Which design is safest for letting a model restart a service?
An allowlisted menu means the model never supplies the command itself.
4. Why must escaping match the destination?
HTML, SQL and shells each treat different characters as commands, so each needs its own handling.
Finish the day
Mark the day complete to save your progress.
Next: Day 10 looks at Excessive Agency, where an agent with too many tools can be hijacked into doing real damage. Today's tool sinks lead straight into it, and you will write rules to limit what an agent may do.
Your notes
Private to this device. Write what you want to remember from today.