SE Bug Bounty Arena / Hunting with AI / Lesson 3
Lesson 3 of 6
Map an AI application before you attack it
You cannot find what an AI feature should not do until you know what it can do. This lesson is how I build that map with an assistant doing the reading.
- 8 min read
- Free
In this lesson
- Start from what is publicRead
- List the inputsRead
- List the powersRead
- Let the application tell youRead
- 3 prompts to copyCopy
- Three-question check+20 XP
Start from what is public
Help pages, API documentation, changelogs and the application's own front-end code tell you most of what a feature does. Paste them in and ask what the AI feature can read and what it can do.
Two questions matter more than any others: where does text enter the model, and where does the model's output go?
List the inputs
The chat box is only one input. Anything the model reads is an input: uploaded files, web pages it fetches, emails it summarises, records it looks up, results from its tools, and memories from earlier sessions.
Ask the assistant to list every one it can find evidence for, and to say which of them a stranger could write to. Those are your routes for indirect prompt injection.
List the powers
Then list what the feature can do: search which data, call which tools, send what, change what. Each power is an impact waiting for a cause.
Put the two lists side by side. An input an outsider controls, connected to a power that matters, is the first thing to test.
Let the application tell you
Often the quickest map comes from asking the feature itself what it can help with. That is normal use and tells you which tools and data it believes it has. Paste the answer back into your assistant and add it to the map.
Prompts to copy
Replace anything in square brackets with your own material.
Below are help pages and API docs for an application with an AI assistant. Using only this material, build two tables: 1. Inputs: every kind of content the model reads (who can write it, and whether an outsider could) 2. Outputs: every place the model's reply is used (shown as text, rendered as HTML, sent somewhere, used to call a tool) Mark each row as "stated" or "inferred". [paste]
From the same material, list what the assistant can do on a user's behalf: data it can search, tools it can call, messages it can send, records it can change. For each, say whether the docs mention a confirmation step.
Combine the two lists. Show me each pair where an input that an outsider can write to connects to a power that affects other users or leaves the system. Rank them by likely impact and tell me what evidence would confirm each one.
Check yourself
Three questions. Sign in to save your progress and earn 20 XP.