Day 9 of 1260 minCipher CadetHands-on lab

Day 9: Improper Output Handling: Never Trust the Model's Answer

Taught by Mr Elite Updated

By the end of today you can

  • Explain why a model's answer must be treated as untrusted input
  • Name the four common sinks: web page, database, shell and other tools
  • Show safely in a test page how a browser can run code hidden in output
  • Choose escaping, validation or an allowlist for each unsafe sink

Warm-up

Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.

Name the three parts of a good finding from yesterday.

What is wrong, why it matters, and how to fix it.

Should an agent remember instructions it found inside untrusted content? Why?

No. Those instructions could be an attacker's, and remembering them would let them steer the agent later.

Picture a support chatbot. A customer asks a question, the model writes a reply, and the app drops that reply straight into the web page. Nobody wrote any attack code. But the customer's message told the model to include a small piece of HTML, the model obliged, and now every agent who opens that ticket runs it in their browser.

The model did what models do. The mistake was the app trusting what came out. Today we follow the answer on its trip through the app and mark every place where that trust is a problem.

The core idea: the answer is untrusted input

Over the last eight days we worried about what goes into a model. Today we flip it and look at what comes out. Here is the rule I want you to carry: the output of a model is untrusted input to whatever system receives it next.

Why untrusted? Because the model's answer is shaped by its prompt, and the prompt can include text from users, web pages, documents and emails. If an attacker can influence any of that, they can influence the answer. The answer is only as trustworthy as the least trustworthy thing the model read.

The OWASP name for getting this wrong is Improper Output Handling. It means the application passes the model's output to another component without checking or cleaning it first.

Let me draw the flow on the whiteboard in plain text. Every arrow after the model is a place where the app has to decide whether to trust what it was handed.

text
customer message --> prompt --> MODEL --> reply --> [ page | database | shell | tool ]
The customer's text goes in on the left. The reply comes out and lands in one or more destinations on the right. Each destination needs its own check.

Notice that the customer never touches the destinations directly. The model sits in the middle and passes their influence along. That is why this bug is easy to miss: the app's own code looks like it is doing all the work.

Sink
Any place the model's answer ends up and gets used: a web page, a database query, a shell command, another tool.
Escaping
Converting special characters so the receiving system treats them as plain text, not as instructions.
Validation
Checking that output has the expected shape, such as a number between 1 and 5, and rejecting anything else.
Allowlist
A short list of permitted values or actions. Anything not on the list is refused.

Quick check

A model's reply is built from a prompt that included a customer's message. How should the app treat the reply?

The four common sinks

A sink is where the answer lands. Each sink has its own way of going wrong, because each one has its own language it understands.

Web page sink
The answer is inserted into HTML. If it contains a script or an event handler, the browser may run it. This is cross-site scripting, often shortened to XSS.
Database sink
The answer is placed into a query. Special characters can change what the query does. This is SQL injection.
Shell sink
The answer becomes part of a command line. Characters like a semicolon can add extra commands.
Tool sink
The answer is passed to another tool or API as its input, such as a refund function or an email sender.

Here is what the risky pattern looks like in code. I am showing these lines so you can recognise them in a review. Do not run them. In each one, the model's text is glued straight into something that has its own command language.

python
page  = "<p>" + reply + "</p>"
query = "SELECT * FROM tickets WHERE dept = '" + reply + "'"
cmd   = "ping -c 1 " + reply
Three unsafe patterns: reply glued into HTML, into SQL, and into a shell command. When you see plus signs or f-strings joining model text to code like this, slow down.

Notice the pattern. In every case, data crosses into a system that has a command language, and the data is allowed to speak that language. Plain text sinks, like a log line shown as text, are much safer.

Watch me do it: see a browser run output

I will show you why a browser rendering output is dangerous, using a harmless page on your own machine. We will use a pretend model reply and a harmless alert box. Nothing here touches any other system, and no internet connection is needed.

html
<div id="out"></div>
<script>
  var reply = '<img src=x onerror="alert('output ran as code')">';
  document.getElementById('out').innerHTML = reply;
</script>
innerHTML tells the browser to treat the string as HTML. The broken image fires its onerror handler, which runs our harmless alert.

Let me explain what just happened. The string looks like an image tag pointing at a file called x, which does not exist. The browser tries to load it, fails, and runs whatever the onerror attribute says. The attacker never wrote a script tag. A plain tag with an event handler was enough.

Now let me fix it. Open the file again and change the word innerHTML to textContent, so the line matches the block below. Save, then refresh the browser tab.

html
document.getElementById('out').textContent = reply;
textContent treats the string as plain text. The browser shows the characters on screen and runs nothing.

After refreshing you should see the raw text of the tag on the page, starting with the less-than sign and the letters img, and no popup. Same reply, different sink behaviour. That one word is the difference between safe and unsafe.

Defences: escaping, validation and allowlists

There are three tools, and each fits a different job. Pick the one that matches the sink.

Let me prove the database point on your own machine. You need Python 3, which includes the sqlite3 module. Create a file called sink.py and paste the code below. It uses a throwaway database that lives only in memory, so nothing is saved anywhere.

python
import sqlite3
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE tickets (id INTEGER, dept TEXT)")
db.execute("INSERT INTO tickets VALUES (1, 'billing'), (2, 'technical')")

reply = "billing' OR '1'='1"

unsafe = "SELECT * FROM tickets WHERE dept = '" + reply + "'"
print(db.execute(unsafe).fetchall())
print(db.execute("SELECT * FROM tickets WHERE dept = ?", (reply,)).fetchall())
The pretend reply carries extra SQL. The first query glues it in as text. The second passes it as a separate value with a question mark placeholder.

Run it with python3 sink.py (or python sink.py on Windows). You should see two lines: [(1, 'billing'), (2, 'technical')] and then []. The first query returned every row because the reply changed the query's meaning. The second returned nothing, because the database treated the whole reply as one odd department name.

One more, for the shell and tool sinks. This snippet never runs anything. It only shows the shape of an allowlist, where the model picks a key and your code owns the command.

python
ACTIONS = {"restart_web": ["systemctl", "restart", "nginx"]}
choice = "restart_web"
if choice in ACTIONS:
    print("would run:", ACTIONS[choice])
else:
    print("refused:", choice)
The model only supplies a key. If the key is not in the dictionary, the request is refused. Try setting choice to 'rm everything' and you will see the refusal.

Quick check

A model picks which department handles a ticket. Which defence fits best?

Lab: Where Does the Answer Go?

Time to apply this. The lab gives you a support app that uses the model's reply in six places. Your job is to decide which sinks are safe and which are not. Log in to your free SecurityElites account and open the lab named in the lab section below.

Work the way I did in the demos. For each place, ask what receives the reply and whether that thing has a command language. Write your judgement down before you check the feedback, so you learn from the ones you get wrong.

Your turn

Where Does the Answer Go?

This lab shows one app with six places where a model's reply is used. It is the best way to practise tracing an answer and judging each sink, because you see the whole picture instead of one example at a time.

Open the lab in a new tab

What to try

What to notice

  • Plain text sinks, like a log shown as text, are low risk
  • The unsafe sinks all hand the answer to something with its own command language
  • Several sinks look harmless until you ask what reads the data next

If it did not work

Assuming our own model is a trusted source

Ask what the model read to produce the answer. If any of it came from users or outside content, treat the reply as untrusted.

Filtering the prompt and ignoring the output

Input filters can be bypassed. Always add checks at the sink, where the damage would happen.

Using one generic cleaner for every sink

Escape for the specific destination. HTML, SQL and shell each need different handling.

Asking the model to 'please be safe' as the only control

Instructions to the model are not enforcement. Use code that escapes, validates and allowlists.

Recap

  • Model output is untrusted input to the next system
  • A sink is anywhere the answer is used: pages, databases, shells, tools
  • A browser given output as HTML may run attacker code
  • Escape for the destination, validate the shape, allowlist the actions
  • Never build shell commands or queries from raw model text
  • Structured output is easier to check than free prose

Cheat sheet

Sink
Where the model's answer is used next
XSS
Output run as code in a browser
textContent
Shows a string as plain text, runs nothing
innerHTML
Treats a string as HTML, risky with untrusted text
Parameterised query
Keeps data separate from SQL commands
Allowlist
Only permitted values pass, the rest are refused

Before the next day

Mission: Map the output sinks

Finish the lab, then turn what you learned into a diagram of one application. Drawing the route of the answer is how reviewers find these bugs in real projects, and it prepares you for the capstone audit.

About 30 min

You are done when: A diagram file day9-sinks.png showing every sink for one application, with unsafe ones marked and a fix noted for each.

Want more? Write one what, why, fix finding for your riskiest sink, in the format from Day 8.

Quiz

4 questions. Your score is saved with your progress.

1. Why is a model's reply untrusted even when you built the app yourself?

2. In the demo, what changed between the unsafe and safe page?

3. Which design is safest for letting a model restart a service?

4. Why must escaping match the destination?

Finish the day

Mark the day complete to save your progress.

Go to Day 10

Next: Day 10 looks at Excessive Agency, where an agent with too many tools can be hijacked into doing real damage. Today's tool sinks lead straight into it, and you will write rules to limit what an agent may do.

Tell someone you are doing this:WhatsAppXLinkedIn

Your notes

Private to this device. Write what you want to remember from today.