Day 11 of 2160 minShell WalkerHands-on lab
Day 11: When Model Output Becomes an Attack: XSS and SQL
Taught by Mr Elite Updated
By the end of today you can
- Explain why model output must be treated as untrusted input
- Prove output-to-HTML XSS with a harmless marker
- Show how text-to-SQL lets a prompt reach tables it should not
- Write a short trust boundary note for each flaw
Warm-up
Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.
In indirect injection, who is the attacker compared with the person using the AI feature?
They can be different people. The attacker plants content, and a victim triggers it by using the feature.
Why did we use harmless markers as evidence when poisoning a knowledge base?
They prove the content was retrieved and obeyed without causing harm, and they are easy to save and repeat.
Picture a chat widget on a shop site. A user types a message, the model replies, and the reply appears on screen with bold text and neat lists. The team wanted it to look like ChatGPT, so they dropped the reply straight into the page as HTML. Nobody asked who controls the words the model writes.
Today we look at the other end of the pipe. Until now, we attacked what goes into the model. Now we look at what comes out and what the application does with it. If that output reaches a browser or a database unchecked, a sentence in a chat box becomes script or SQL.
Model output is untrusted input
A trust boundary is the line where data passes from something you do not control into something you do. In classic web testing, the browser sends data and the server must not trust it. With an AI feature there is a second boundary: the model sits between the user and your code, and it can be steered by the user, by documents (Day 10), or by a poisoned knowledge base.
So the model's reply is just another form of user input. You already know the rules from Day 6. Untrusted text placed into HTML must be escaped or sanitised. Untrusted text placed into a query must be parameterised or tightly restricted. The model does not change those rules. It only hides the fact that a human influenced the string.
Here is the usual chain. A person writes a message. The model turns it into a reply or a query. The application takes that string and hands it to a powerful interpreter: the browser's HTML parser, or the database engine. The flaw is almost never the model. It is the missing check between the model and the interpreter.
- innerHTML
- A browser property that parses a string as HTML. Any script-bearing tag inside it can run.
- Output encoding
- Turning characters like < and > into harmless text such as < and > before display.
- Text-to-SQL
- A feature where a model turns a plain-English question into a SQL query that the app then runs.
- Least privilege
- Giving a component only the access it needs, here a database account that can read only the shop's own data.
Quick check
A chatbot reply is inserted into the page with innerHTML. Where is the failure?
The flaw is in the application. Model output crossed a trust boundary without being encoded or sanitised.
Watch me do it: both flaws on your own machine
Before the labs, I want you to see both bugs with no model involved. We will fake the model reply with a fixed string. This runs only on your own computer, in a new folder. You need Python 3 and a browser. Check Python first by running python3 --version. On Windows the command is usually python. You should see a version number starting with 3.
mkdir day11 && cd day11import html
reply = "Here is **bold** <img src=x onerror=alert('model-output')>"
bad = "<div id='chat'>" + reply + "</div>"
good = "<div id='chat'>" + html.escape(reply) + "</div>"
open('bad.html', 'w').write(bad)
open('good.html', 'w').write(good)What happened: the browser parsed the image tag in bad.html, the image failed to load because src=x is not a real file, and the onerror handler ran. In good.html, html.escape turned the angle brackets into text, so the browser showed the tag instead of obeying it. That one function call is the whole defence at this layer. If bad.html showed no alert, check your browser did not block it, and view the page source to confirm the img tag is there.
import sqlite3
db = sqlite3.connect(':memory:')
db.executescript("create table orders(id int, shop text, item text); create table users(name text, secret text); insert into orders values (1,'mine','mug'); insert into users values ('admin','demo-secret');")
def fake_model(q):
if 'all accounts' in q:
return 'SELECT * FROM users'
return "SELECT * FROM orders WHERE shop='mine'"
for q in ['show my orders', 'show all accounts']:
sql = fake_model(q)
print(q, '->', sql)
print(db.execute(sql).fetchall())show my orders -> SELECT * FROM orders WHERE shop='mine'
[(1, 'mine', 'mug')]
show all accounts -> SELECT * FROM users
[('admin', 'demo-secret')]Read it line by line. The first question returns the mug order, as intended. The second returns the users table, including demo-secret. The app never checked the SQL. It trusted the model to stay in its lane, and a different question moved it out.
Quick check
Which is the stronger fix for the text-to-SQL flaw?
Database permissions are enforced regardless of what the model writes. Prompts can be talked around.
Fix both flaws and see the difference
Proving a bug is half the job. You also need to know what a correct fix looks like, so your report can recommend one. We already fixed the XSS demo with html.escape. In a live page the browser-side equivalent is textContent, which always inserts plain text.
// unsafe: parses the reply as HTML
chat.innerHTML = reply;
// safe: shows the reply as plain text
chat.textContent = reply;Now the SQL side. Make a copy of demo_sql.py called demo_sql_fixed.py. Remove the users table and its insert from the executescript string, so the connection holds only orders. Then wrap the execute line so errors are printed. This mimics a database account that simply cannot see other tables.
import sqlite3
db = sqlite3.connect(':memory:')
db.executescript("create table orders(id int, shop text, item text); insert into orders values (1,'mine','mug');")
def fake_model(q):
if 'all accounts' in q:
return 'SELECT * FROM users'
return "SELECT * FROM orders WHERE shop='mine'"
for q in ['show my orders', 'show all accounts']:
sql = fake_model(q)
print(q, '->', sql)
try:
print(db.execute(sql).fetchall())
except sqlite3.OperationalError as e:
print('blocked:', e)Run it. The first question still works. The second prints blocked: no such table: users. The model still wrote the dangerous query. It simply could not do any harm. That is what a real control looks like: it works whatever the model says.
Quick check
In the fixed demo, why does the query for users fail?
The restriction sits in the database, not in the prompt or the model, so it holds even when the model misbehaves.
The labs: where the boundary sits
Today has two practice labs, both on the SecurityElites platform. Only test them there. First is ai-llm-output-xss-1, ChatBotApp, which renders the model's reply as innerHTML. Second is ai-text-to-sql-injection-1, DataPilot, which turns questions into SQL and runs them against a replica holding every table. Both are rated advanced, so take your time.
For each lab, ask the same three questions. Where does my text enter? What does the model turn it into? Where does that output land, and does anything check it on the way? Write the answers in a notes file as you go. Those notes become today's mission.
Work in small steps. In ChatBotApp, start with a plain message and look at how the reply is built in the page before you try anything clever. In DataPilot, start with a question about your own orders and see whether the generated SQL is shown. Seeing normal behaviour first tells you what abnormal looks like.
Write the evidence, not just the win
A flaw you cannot explain is hard to report. Day 13 is about reports, so start the habit now. For each lab, save three things: the exact message you sent, what the model returned, and what the application did with it. Add a screenshot or copied output.
Then name the boundary. In the XSS lab, it sat between model reply and the browser's DOM. In the SQL lab, it sat between model-written SQL and the database. Your job in the paragraph is to say what should have been there, such as encoding, sanitising, an allow-list, or least privilege, and why it was absent.
Message sent: (exact text)
Model returned: (exact reply or SQL)
Application did: (what the page or database did)
Boundary: (from where to where)
Missing check: (what should have stopped it)
Fix: (the enforced control)Your turn
AI LLM-Output XSS
Start with ChatBotApp because the flaw is easier to see: the model's reply is rendered as HTML. Once you have proved it, repeat the same thinking on the DataPilot lab (ai-text-to-sql-injection-1), where the output lands in a database instead of a page.
What to try
What to notice
- Whether the app escapes the reply, sanitises it, or inserts it raw
- Whether the model or the application is the component that actually fails
- What data the database account could see, compared with what it should see
- How rewording a request changes what the model produces
If it did not work
Blaming the model and writing 'the AI said something bad' as the finding
Name the application's failure: unescaped output or unrestricted SQL execution. The model is only the delivery route.
Giving up when the model refuses a payload
Reword as a formatting or example request, and use a harmless marker. Refusals vary by wording and run to run.
Using a real-looking payload that could cause damage
Stick to alerts, visible text and read-only queries. Evidence of execution is enough.
Recommending a stricter prompt as the fix
Prompts are not controls. Recommend output encoding or sanitising for XSS and least-privilege database access for SQL.
Recap
- Model output is untrusted input, because users and documents can steer it
- Rendering model output with innerHTML without sanitising leads to XSS
- Text-to-SQL runs model-written queries, so the database must limit what they can reach
- Prompt instructions are requests, while encoding and permissions are enforced controls
- Always name the trust boundary and where its check was missing
- Use harmless markers and read-only proof, and save the exact input and output
Cheat sheet
innerHTML- Parses a string as HTML, so unsafe for untrusted text
textContent- Inserts text as plain text, the safer alternative
html.escape()- Python function that encodes < > & and quotes
Parameterised query- Keeps data separate from SQL code
Least privilege- Database account limited to the data the feature needs
Trust boundary note- One paragraph: where the boundary sat and where the check failed
Before the next day
Mission: Solve both labs and name the boundary
Complete ChatBotApp and DataPilot, then write one paragraph per lab explaining where the trust boundary sat and where it failed. This builds the evidence habit you need for report writing on Day 13.
About 35 min
You are done when: A day11 folder with two notes files, each containing evidence and a trust boundary paragraph, plus two new rows in bug-class-table.md.
Want more? Rewrite demo_xss.py so it renders safely with a sanitising approach that allows bold text but blocks tags with event handlers, and describe why a simple escape loses the formatting.
Quiz
4 questions. Your score is saved with your progress.
1. Why is a model's reply treated as untrusted input?
Anyone who can steer the model can shape its output, so the application must handle it like any other untrusted data.
2. In demo bad.html, what made the alert fire?
The raw string was placed into the page as markup, so the browser treated the tag as real HTML.
3. DataPilot is told to query only your shop's orders but can read other tables. What is the root problem?
Excess database privilege means any query the model writes can reach everything. Least privilege fixes it at the right layer.
4. Which finding statement is strongest for a report?
It names the component, the mechanism and the impact, which is what a triager can reproduce and act on.
Finish the day
Mark the day complete to save your progress.
Next: Day 12 is a module review where you test an AI product for access control and SSRF problems. Today's boundary thinking and your Day 10 and 11 notes feed straight into it.
Your notes
Private to this device. Write what you want to remember from today.