Day 3 of 2160 minShell WalkerHands-on lab
Day 3: Trust but Verify: Catching AI False Confidence
Taught by Mr Elite Updated
By the end of today you can
- Spot the common shapes of confident but wrong AI answers about security behaviour
- Turn any assistant claim into a test you can run with curl against your own server
- Use a prompt template that makes the assistant state its assumptions and confidence up front
- Record claims, tests and results in a table you can trust
Warm-up
Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.
Why does safe harbour in a bounty policy only protect you part of the time?
It only applies while you stay inside the policy and its scope rules.
What is the point of using placeholders like when you share a request with an assistant?
They keep the shape of the problem so the assistant can help, while the real secret never leaves your machine.
Picture this. You ask an assistant how to stop a session cookie being read by JavaScript, and it replies in a calm, tidy paragraph. It names a flag, tells you the default value, and cites a CVE number for good measure. You paste the advice into your report notes. Later a triager tells you the flag does not exist, and the CVE belongs to a printer driver.
Assistants write fluent text whether or not they are right. Today I will show you how to treat every answer as a claim to be tested, not a fact to be copied. You will run a small local server, check claims with curl, and then meet a lab where an AI wrote tests that look reassuring but prove nothing.
Why confident answers go wrong
A language model predicts likely text. It does not look things up and it does not run anything unless you give it a tool to do so. So a wrong answer sounds exactly as sure as a right one. Building on yesterday, where we kept secrets out of prompts, today we protect the other direction: keeping bad information out of your notes and reports.
In security work the same few failures turn up again and again. Learn the shapes and you will spot them quickly.
- Hallucination
- A made-up detail stated as fact, such as a flag, endpoint or function that does not exist.
- CVE
- A public ID for a known vulnerability, written like CVE-2021-44228. The number is easy to invent and easy to check.
- Claim
- One specific statement you can test, such as 'HttpOnly stops JavaScript reading the cookie'.
- Assumption
- Something the answer depends on but never says out loud, such as the browser, framework or version.
Here is what a hallucination looks like in practice. Suppose an assistant tells you to add a cookie attribute called HttpOnlyStrict. It sounds plausible because it is built from real words. But no browser knows it, and a browser simply ignores attributes it does not recognise. Your cookie would stay readable and you would never see an error. That silence is what makes this failure dangerous.
Checking a CVE takes about a minute and needs no account. Do it every time, because a number that looks precise is the easiest thing for a model to invent.
Quick check
An assistant gives you a CVE number that supports its point. What should you do first?
Asking the same model if it is sure only gets you more fluent text; an independent source is what confirms a CVE.
A prompt template that asks for assumptions up front
You can lower the odds of a bad answer by changing the question. I want the assistant to list what it is assuming, split facts from guesses, and tell me how to test each point. Save this in your workbench folder as verify-prompt.md.
Question: <your technical question>
Before answering:
1. List your assumptions (browser, framework, version).
2. Number each claim you make.
3. Mark each claim HIGH, MEDIUM or LOW confidence.
4. For every claim, give one curl command or browser check I can run on my own local server to test it.
5. If you are not sure a flag, header or CVE exists, say so instead of guessing.This does not make the answer correct. It makes the answer testable, which is the real goal. Numbered claims are easy to copy into a table, and the confidence labels tell you where to look first.
When you get a reply, read it in this order. First the assumptions: are they sensible for your setup? Then the LOW and MEDIUM claims, since they are the likeliest to be wrong. A good reply may also say 'I am not sure this exists'. That honesty is the best outcome the template can produce, so do not treat it as a failure.
Watch me do it: test a cookie claim with curl
We will use a tiny Python server on your own machine. Nothing here touches a third-party system. I will set a cookie with a few flags, then check what the assistant claims against what the server really sends. You need Python 3 and curl, both free. Run python3 --version first. You should see something like Python 3.10.12. Any 3.x is fine.
from http.server import BaseHTTPRequestHandler, HTTPServer
class H(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header('Set-Cookie', 'sid=abc123; HttpOnly; SameSite=Lax; Path=/')
self.send_header('Content-Type', 'text/html')
self.end_headers()
self.wfile.write(b'<h1>hello</h1>')
print('Serving on http://127.0.0.1:8081')
HTTPServer(('127.0.0.1', 8081), H).serve_forever()curl -i http://127.0.0.1:8081/The full output should look roughly like this, with your own date and Python version. The line that matters is Set-Cookie.
HTTP/1.0 200 OK
Server: BaseHTTP/0.6 Python/3.10.12
Date: Mon, 01 Jan 2024 10:00:00 GMT
Set-Cookie: sid=abc123; HttpOnly; SameSite=Lax; Path=/
Content-Type: text/html
<h1>hello</h1>Notice what this proved and what it did not. curl shows the server sent the flags. It cannot show how a browser enforces them, because curl ignores HttpOnly entirely. If an assistant claims 'HttpOnly blocks document.cookie', the right test is the browser console.
Some claims cannot be tested here at all. The Secure flag only matters over HTTPS, and our server speaks plain HTTP. SameSite defaults also differ between browsers and versions. Label those claims 'untestable here' in your table. That is an honest result, and far better than guessing.
Quick check
Why can curl not confirm that HttpOnly hides a cookie from JavaScript?
HttpOnly is enforced by browsers, so you need the browser console to test that claim.
Record claim, test, result
A claim table keeps you honest. For each numbered claim write what the assistant said, the exact test you ran, and the outcome: confirmed, wrong, or untestable here. Compare both assistants side by side, because disagreements point straight at the claims worth testing first.
| # | Assistant | Claim | Test | Result |
|---|-----------|-------|------|--------|
| 1 | Claude | HttpOnly hides cookie from JS | document.cookie in console | Confirmed |
| 2 | ChatGPT | <claim> | <command> | <result> |Here is how a few more rows might read once filled in. Row 3: 'The server sends the Set-Cookie header', test curl -i, result confirmed. Row 4: 'Secure cookies are only sent over HTTPS', test none possible on our HTTP server, result untestable here. Row 5: 'SameSite=Lax is required for cookies to work', test remove the flag and reload, result wrong if the cookie still arrives.
Write the result straight after you run the test, while the output is on screen. If you leave it until later you will remember what you expected rather than what you saw. Keep the Test column exact enough that someone else could repeat it.
Lab: Tests That Test Nothing
The same habit applies to code. In the lab ai-dev-fake-tests-1, an assistant wrote seven tests for a discount function and coverage hit 100 percent. Some of those tests would still pass if the function were broken. Your job is to find them. Open the lab from your SecurityElites dashboard and read the tests before you read my hints.
Work through it in this order. Read the discount function first so you know what it should do. Then take the tests one at a time and write a one-line answer to 'what would make this fail?'. If you cannot think of anything, you have probably found a test that checks nothing. The lab is a practice environment made for this, so you can break the code freely.
Your turn
Tests That Test Nothing
This lab shows false confidence in a different form: tests that look thorough but check nothing. It trains the question I want you to ask every time, which is what result would prove this wrong. Take your time reading the code before you judge any test.
What to try
What to notice
- High coverage tells you code ran, not that results were checked.
- Weak tests often look tidy and have confident names.
- The fix is usually a specific expected value, not more tests.
If it did not work
Copying a CVE number, flag or endpoint from an answer straight into notes
Look it up in an independent source, such as the official CVE database or the tool's own help output, before it goes anywhere.
Asking the same assistant 'are you sure?' and accepting yes
Run a test. Self-confirmation from a model is not evidence.
Testing a browser behaviour with curl only
Match the test to the layer: curl for what the server sends, browser tools for what the browser does.
Pasting real target data into the verification prompt
Use your local server and placeholders, as in Day 2.
Forgetting to restart the server after editing cookie_server.py
Press Ctrl+C in the server terminal and run python3 cookie_server.py again. Then re-run curl and check the header really changed.
Recap
- Assistants sound equally sure when right and when wrong
- Invented flags, endpoints and CVE numbers are the most common failures
- Ask for assumptions, numbered claims, confidence labels and a test for each
- Every claim needs a test you ran yourself, at the right layer
- Agreement between two assistants is not proof
- Passing tests prove nothing unless they could fail
Cheat sheet
curl -i URL- Show response headers and body to check what the server really sent
document.cookie- Browser console check for which cookies JavaScript can see
HttpOnly- Cookie flag that stops JavaScript reading the cookie, enforced by browsers
SameSite=Lax- Cookie flag that limits cross-site sending, enforced by browsers
Claim table- Claim, test, result for each statement you rely on
verify-prompt.md- Template asking for assumptions, numbered claims, confidence and tests
Before the next day
Mission: Claim, test, result table
Ask both assistants the same question about a header or cookie flag, then test what they say against your local server. The point is to practise treating answers as claims, and to see where the two assistants differ. Everything runs on your own machine.
About 35 min
You are done when: claims-table.md with at least five tested claims from two assistants, plus verify-prompt.md in your workbench folder.
Want more? Find one claim neither assistant could be tested on locally and write down what extra setup, such as HTTPS with a self-signed certificate, would let you test it.
Quiz
4 questions. Your score is saved with your progress.
1. An assistant says a header is 'Secure-Content-Policy' and gives its default value. What is the best first move?
The name may be invented. Independent documentation or a real response is the way to confirm it exists.
2. Why does the template ask for a test command with each claim?
A testable claim can be shown wrong quickly. The template does not remove errors, it exposes them.
3. Both assistants agree that a cookie flag behaves a certain way. What does that tell you?
Agreement raises plausibility only. Models can share the same mistake, so you still run your own test.
4. In the fake tests lab, why is 100 percent coverage not good evidence?
Coverage measures execution. A test with no meaningful assertion executes code and still proves nothing.
Finish the day
Mark the day complete to save your progress.
Next: Day 4 is a module review. You will take a scope statement and build a verified, AI-assisted recon plan, using today's claim table habit and Day 2's redaction rules together.
Your notes
Private to this device. Write what you want to remember from today.