Day 11 of 2160 minShell WalkerHands-on lab

Day 11: When Model Output Becomes an Attack: XSS and SQL

Taught by Mr Elite Updated

By the end of today you can

  • Explain why model output must be treated as untrusted input
  • Prove output-to-HTML XSS with a harmless marker
  • Show how text-to-SQL lets a prompt reach tables it should not
  • Write a short trust boundary note for each flaw

Warm-up

Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.

In indirect injection, who is the attacker compared with the person using the AI feature?

They can be different people. The attacker plants content, and a victim triggers it by using the feature.

Why did we use harmless markers as evidence when poisoning a knowledge base?

They prove the content was retrieved and obeyed without causing harm, and they are easy to save and repeat.

Picture a chat widget on a shop site. A user types a message, the model replies, and the reply appears on screen with bold text and neat lists. The team wanted it to look like ChatGPT, so they dropped the reply straight into the page as HTML. Nobody asked who controls the words the model writes.

Today we look at the other end of the pipe. Until now, we attacked what goes into the model. Now we look at what comes out and what the application does with it. If that output reaches a browser or a database unchecked, a sentence in a chat box becomes script or SQL.

Model output is untrusted input

A trust boundary is the line where data passes from something you do not control into something you do. In classic web testing, the browser sends data and the server must not trust it. With an AI feature there is a second boundary: the model sits between the user and your code, and it can be steered by the user, by documents (Day 10), or by a poisoned knowledge base.

So the model's reply is just another form of user input. You already know the rules from Day 6. Untrusted text placed into HTML must be escaped or sanitised. Untrusted text placed into a query must be parameterised or tightly restricted. The model does not change those rules. It only hides the fact that a human influenced the string.

Here is the usual chain. A person writes a message. The model turns it into a reply or a query. The application takes that string and hands it to a powerful interpreter: the browser's HTML parser, or the database engine. The flaw is almost never the model. It is the missing check between the model and the interpreter.

innerHTML
A browser property that parses a string as HTML. Any script-bearing tag inside it can run.
Output encoding
Turning characters like < and > into harmless text such as < and > before display.
Text-to-SQL
A feature where a model turns a plain-English question into a SQL query that the app then runs.
Least privilege
Giving a component only the access it needs, here a database account that can read only the shop's own data.

Quick check

A chatbot reply is inserted into the page with innerHTML. Where is the failure?

Watch me do it: both flaws on your own machine

Before the labs, I want you to see both bugs with no model involved. We will fake the model reply with a fixed string. This runs only on your own computer, in a new folder. You need Python 3 and a browser. Check Python first by running python3 --version. On Windows the command is usually python. You should see a version number starting with 3.

bash
mkdir day11 && cd day11
Creates a working folder for today's demos and moves into it.
python
import html
reply = "Here is **bold** <img src=x onerror=alert('model-output')>"
bad = "<div id='chat'>" + reply + "</div>"
good = "<div id='chat'>" + html.escape(reply) + "</div>"
open('bad.html', 'w').write(bad)
open('good.html', 'w').write(good)
Same pretend reply, two renderings. bad.html inserts it raw. good.html escapes it first. The alert is a harmless marker, not an attack.

What happened: the browser parsed the image tag in bad.html, the image failed to load because src=x is not a real file, and the onerror handler ran. In good.html, html.escape turned the angle brackets into text, so the browser showed the tag instead of obeying it. That one function call is the whole defence at this layer. If bad.html showed no alert, check your browser did not block it, and view the page source to confirm the img tag is there.

python
import sqlite3
db = sqlite3.connect(':memory:')
db.executescript("create table orders(id int, shop text, item text); create table users(name text, secret text); insert into orders values (1,'mine','mug'); insert into users values ('admin','demo-secret');")
def fake_model(q):
    if 'all accounts' in q:
        return 'SELECT * FROM users'
    return "SELECT * FROM orders WHERE shop='mine'"
for q in ['show my orders', 'show all accounts']:
    sql = fake_model(q)
    print(q, '->', sql)
    print(db.execute(sql).fetchall())
A stand-in for text-to-SQL. The app runs whatever SQL the 'model' returns against a database holding every table.
text
show my orders -> SELECT * FROM orders WHERE shop='mine'
[(1, 'mine', 'mug')]
show all accounts -> SELECT * FROM users
[('admin', 'demo-secret')]
The output you should see. The second question reaches the users table.

Read it line by line. The first question returns the mug order, as intended. The second returns the users table, including demo-secret. The app never checked the SQL. It trusted the model to stay in its lane, and a different question moved it out.

Quick check

Which is the stronger fix for the text-to-SQL flaw?

Fix both flaws and see the difference

Proving a bug is half the job. You also need to know what a correct fix looks like, so your report can recommend one. We already fixed the XSS demo with html.escape. In a live page the browser-side equivalent is textContent, which always inserts plain text.

javascript
// unsafe: parses the reply as HTML
chat.innerHTML = reply;
// safe: shows the reply as plain text
chat.textContent = reply;
The two lines differ by one property name. Only the second keeps tags from running.

Now the SQL side. Make a copy of demo_sql.py called demo_sql_fixed.py. Remove the users table and its insert from the executescript string, so the connection holds only orders. Then wrap the execute line so errors are printed. This mimics a database account that simply cannot see other tables.

python
import sqlite3
db = sqlite3.connect(':memory:')
db.executescript("create table orders(id int, shop text, item text); insert into orders values (1,'mine','mug');")
def fake_model(q):
    if 'all accounts' in q:
        return 'SELECT * FROM users'
    return "SELECT * FROM orders WHERE shop='mine'"
for q in ['show my orders', 'show all accounts']:
    sql = fake_model(q)
    print(q, '->', sql)
    try:
        print(db.execute(sql).fetchall())
    except sqlite3.OperationalError as e:
        print('blocked:', e)
Same fake model, but the connection only holds the data the feature needs.

Run it. The first question still works. The second prints blocked: no such table: users. The model still wrote the dangerous query. It simply could not do any harm. That is what a real control looks like: it works whatever the model says.

Quick check

In the fixed demo, why does the query for users fail?

The labs: where the boundary sits

Today has two practice labs, both on the SecurityElites platform. Only test them there. First is ai-llm-output-xss-1, ChatBotApp, which renders the model's reply as innerHTML. Second is ai-text-to-sql-injection-1, DataPilot, which turns questions into SQL and runs them against a replica holding every table. Both are rated advanced, so take your time.

For each lab, ask the same three questions. Where does my text enter? What does the model turn it into? Where does that output land, and does anything check it on the way? Write the answers in a notes file as you go. Those notes become today's mission.

Work in small steps. In ChatBotApp, start with a plain message and look at how the reply is built in the page before you try anything clever. In DataPilot, start with a question about your own orders and see whether the generated SQL is shown. Seeing normal behaviour first tells you what abnormal looks like.

Write the evidence, not just the win

A flaw you cannot explain is hard to report. Day 13 is about reports, so start the habit now. For each lab, save three things: the exact message you sent, what the model returned, and what the application did with it. Add a screenshot or copied output.

Then name the boundary. In the XSS lab, it sat between model reply and the browser's DOM. In the SQL lab, it sat between model-written SQL and the database. Your job in the paragraph is to say what should have been there, such as encoding, sanitising, an allow-list, or least privilege, and why it was absent.

text
Message sent: (exact text)
Model returned: (exact reply or SQL)
Application did: (what the page or database did)
Boundary: (from where to where)
Missing check: (what should have stopped it)
Fix: (the enforced control)
A six-line template for each notes file. Fill every line before you call a lab finished.

Your turn

AI LLM-Output XSS

Start with ChatBotApp because the flaw is easier to see: the model's reply is rendered as HTML. Once you have proved it, repeat the same thinking on the DataPilot lab (ai-text-to-sql-injection-1), where the output lands in a database instead of a page.

Open the lab in a new tab

What to try

What to notice

  • Whether the app escapes the reply, sanitises it, or inserts it raw
  • Whether the model or the application is the component that actually fails
  • What data the database account could see, compared with what it should see
  • How rewording a request changes what the model produces

If it did not work

Blaming the model and writing 'the AI said something bad' as the finding

Name the application's failure: unescaped output or unrestricted SQL execution. The model is only the delivery route.

Giving up when the model refuses a payload

Reword as a formatting or example request, and use a harmless marker. Refusals vary by wording and run to run.

Using a real-looking payload that could cause damage

Stick to alerts, visible text and read-only queries. Evidence of execution is enough.

Recommending a stricter prompt as the fix

Prompts are not controls. Recommend output encoding or sanitising for XSS and least-privilege database access for SQL.

Recap

  • Model output is untrusted input, because users and documents can steer it
  • Rendering model output with innerHTML without sanitising leads to XSS
  • Text-to-SQL runs model-written queries, so the database must limit what they can reach
  • Prompt instructions are requests, while encoding and permissions are enforced controls
  • Always name the trust boundary and where its check was missing
  • Use harmless markers and read-only proof, and save the exact input and output

Cheat sheet

innerHTML
Parses a string as HTML, so unsafe for untrusted text
textContent
Inserts text as plain text, the safer alternative
html.escape()
Python function that encodes < > & and quotes
Parameterised query
Keeps data separate from SQL code
Least privilege
Database account limited to the data the feature needs
Trust boundary note
One paragraph: where the boundary sat and where the check failed

Before the next day

Mission: Solve both labs and name the boundary

Complete ChatBotApp and DataPilot, then write one paragraph per lab explaining where the trust boundary sat and where it failed. This builds the evidence habit you need for report writing on Day 13.

About 35 min

You are done when: A day11 folder with two notes files, each containing evidence and a trust boundary paragraph, plus two new rows in bug-class-table.md.

Want more? Rewrite demo_xss.py so it renders safely with a sanitising approach that allows bold text but blocks tags with event handlers, and describe why a simple escape loses the formatting.

Quiz

4 questions. Your score is saved with your progress.

1. Why is a model's reply treated as untrusted input?

2. In demo bad.html, what made the alert fire?

3. DataPilot is told to query only your shop's orders but can read other tables. What is the root problem?

4. Which finding statement is strongest for a report?

Finish the day

Mark the day complete to save your progress.

Go to Day 12

Next: Day 12 is a module review where you test an AI product for access control and SSRF problems. Today's boundary thinking and your Day 10 and 11 notes feed straight into it.

Tell someone you are doing this:WhatsAppXLinkedIn

Your notes

Private to this device. Write what you want to remember from today.