Day 3 of 2160 minShell WalkerHands-on lab

Day 3: Trust but Verify: Catching AI False Confidence

Taught by Mr Elite Updated

By the end of today you can

  • Spot the common shapes of confident but wrong AI answers about security behaviour
  • Turn any assistant claim into a test you can run with curl against your own server
  • Use a prompt template that makes the assistant state its assumptions and confidence up front
  • Record claims, tests and results in a table you can trust

Warm-up

Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.

Why does safe harbour in a bounty policy only protect you part of the time?

It only applies while you stay inside the policy and its scope rules.

What is the point of using placeholders like when you share a request with an assistant?

They keep the shape of the problem so the assistant can help, while the real secret never leaves your machine.

Picture this. You ask an assistant how to stop a session cookie being read by JavaScript, and it replies in a calm, tidy paragraph. It names a flag, tells you the default value, and cites a CVE number for good measure. You paste the advice into your report notes. Later a triager tells you the flag does not exist, and the CVE belongs to a printer driver.

Assistants write fluent text whether or not they are right. Today I will show you how to treat every answer as a claim to be tested, not a fact to be copied. You will run a small local server, check claims with curl, and then meet a lab where an AI wrote tests that look reassuring but prove nothing.

Why confident answers go wrong

A language model predicts likely text. It does not look things up and it does not run anything unless you give it a tool to do so. So a wrong answer sounds exactly as sure as a right one. Building on yesterday, where we kept secrets out of prompts, today we protect the other direction: keeping bad information out of your notes and reports.

In security work the same few failures turn up again and again. Learn the shapes and you will spot them quickly.

Hallucination
A made-up detail stated as fact, such as a flag, endpoint or function that does not exist.
CVE
A public ID for a known vulnerability, written like CVE-2021-44228. The number is easy to invent and easy to check.
Claim
One specific statement you can test, such as 'HttpOnly stops JavaScript reading the cookie'.
Assumption
Something the answer depends on but never says out loud, such as the browser, framework or version.

Here is what a hallucination looks like in practice. Suppose an assistant tells you to add a cookie attribute called HttpOnlyStrict. It sounds plausible because it is built from real words. But no browser knows it, and a browser simply ignores attributes it does not recognise. Your cookie would stay readable and you would never see an error. That silence is what makes this failure dangerous.

Checking a CVE takes about a minute and needs no account. Do it every time, because a number that looks precise is the easiest thing for a model to invent.

Quick check

An assistant gives you a CVE number that supports its point. What should you do first?

A prompt template that asks for assumptions up front

You can lower the odds of a bad answer by changing the question. I want the assistant to list what it is assuming, split facts from guesses, and tell me how to test each point. Save this in your workbench folder as verify-prompt.md.

text
Question: <your technical question>

Before answering:
1. List your assumptions (browser, framework, version).
2. Number each claim you make.
3. Mark each claim HIGH, MEDIUM or LOW confidence.
4. For every claim, give one curl command or browser check I can run on my own local server to test it.
5. If you are not sure a flag, header or CVE exists, say so instead of guessing.
Reusable verification prompt. Paste your question in the first line. No secrets or real target data go in.

This does not make the answer correct. It makes the answer testable, which is the real goal. Numbered claims are easy to copy into a table, and the confidence labels tell you where to look first.

When you get a reply, read it in this order. First the assumptions: are they sensible for your setup? Then the LOW and MEDIUM claims, since they are the likeliest to be wrong. A good reply may also say 'I am not sure this exists'. That honesty is the best outcome the template can produce, so do not treat it as a failure.

Watch me do it: test a cookie claim with curl

We will use a tiny Python server on your own machine. Nothing here touches a third-party system. I will set a cookie with a few flags, then check what the assistant claims against what the server really sends. You need Python 3 and curl, both free. Run python3 --version first. You should see something like Python 3.10.12. Any 3.x is fine.

python
from http.server import BaseHTTPRequestHandler, HTTPServer

class H(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header('Set-Cookie', 'sid=abc123; HttpOnly; SameSite=Lax; Path=/')
        self.send_header('Content-Type', 'text/html')
        self.end_headers()
        self.wfile.write(b'<h1>hello</h1>')

print('Serving on http://127.0.0.1:8081')
HTTPServer(('127.0.0.1', 8081), H).serve_forever()
A local server that sets one cookie with HttpOnly and SameSite=Lax. It binds to 127.0.0.1 so only your machine can reach it.
bash
curl -i http://127.0.0.1:8081/
The -i flag prints response headers. Look for: Set-Cookie: sid=abc123; HttpOnly; SameSite=Lax; Path=/

The full output should look roughly like this, with your own date and Python version. The line that matters is Set-Cookie.

text
HTTP/1.0 200 OK
Server: BaseHTTP/0.6 Python/3.10.12
Date: Mon, 01 Jan 2024 10:00:00 GMT
Set-Cookie: sid=abc123; HttpOnly; SameSite=Lax; Path=/
Content-Type: text/html

<h1>hello</h1>
Example curl output. The cookie line proves what the server sent, nothing more.

Notice what this proved and what it did not. curl shows the server sent the flags. It cannot show how a browser enforces them, because curl ignores HttpOnly entirely. If an assistant claims 'HttpOnly blocks document.cookie', the right test is the browser console.

Some claims cannot be tested here at all. The Secure flag only matters over HTTPS, and our server speaks plain HTTP. SameSite defaults also differ between browsers and versions. Label those claims 'untestable here' in your table. That is an honest result, and far better than guessing.

Quick check

Why can curl not confirm that HttpOnly hides a cookie from JavaScript?

Record claim, test, result

A claim table keeps you honest. For each numbered claim write what the assistant said, the exact test you ran, and the outcome: confirmed, wrong, or untestable here. Compare both assistants side by side, because disagreements point straight at the claims worth testing first.

text
| # | Assistant | Claim | Test | Result |
|---|-----------|-------|------|--------|
| 1 | Claude | HttpOnly hides cookie from JS | document.cookie in console | Confirmed |
| 2 | ChatGPT | <claim> | <command> | <result> |
Starter table for claims-table.md. Keep the real command in the Test column so you can repeat it.

Here is how a few more rows might read once filled in. Row 3: 'The server sends the Set-Cookie header', test curl -i, result confirmed. Row 4: 'Secure cookies are only sent over HTTPS', test none possible on our HTTP server, result untestable here. Row 5: 'SameSite=Lax is required for cookies to work', test remove the flag and reload, result wrong if the cookie still arrives.

Write the result straight after you run the test, while the output is on screen. If you leave it until later you will remember what you expected rather than what you saw. Keep the Test column exact enough that someone else could repeat it.

Lab: Tests That Test Nothing

The same habit applies to code. In the lab ai-dev-fake-tests-1, an assistant wrote seven tests for a discount function and coverage hit 100 percent. Some of those tests would still pass if the function were broken. Your job is to find them. Open the lab from your SecurityElites dashboard and read the tests before you read my hints.

Work through it in this order. Read the discount function first so you know what it should do. Then take the tests one at a time and write a one-line answer to 'what would make this fail?'. If you cannot think of anything, you have probably found a test that checks nothing. The lab is a practice environment made for this, so you can break the code freely.

Your turn

Tests That Test Nothing

This lab shows false confidence in a different form: tests that look thorough but check nothing. It trains the question I want you to ask every time, which is what result would prove this wrong. Take your time reading the code before you judge any test.

Open the lab in a new tab

What to try

What to notice

  • High coverage tells you code ran, not that results were checked.
  • Weak tests often look tidy and have confident names.
  • The fix is usually a specific expected value, not more tests.

If it did not work

Copying a CVE number, flag or endpoint from an answer straight into notes

Look it up in an independent source, such as the official CVE database or the tool's own help output, before it goes anywhere.

Asking the same assistant 'are you sure?' and accepting yes

Run a test. Self-confirmation from a model is not evidence.

Testing a browser behaviour with curl only

Match the test to the layer: curl for what the server sends, browser tools for what the browser does.

Pasting real target data into the verification prompt

Use your local server and placeholders, as in Day 2.

Forgetting to restart the server after editing cookie_server.py

Press Ctrl+C in the server terminal and run python3 cookie_server.py again. Then re-run curl and check the header really changed.

Recap

  • Assistants sound equally sure when right and when wrong
  • Invented flags, endpoints and CVE numbers are the most common failures
  • Ask for assumptions, numbered claims, confidence labels and a test for each
  • Every claim needs a test you ran yourself, at the right layer
  • Agreement between two assistants is not proof
  • Passing tests prove nothing unless they could fail

Cheat sheet

curl -i URL
Show response headers and body to check what the server really sent
document.cookie
Browser console check for which cookies JavaScript can see
HttpOnly
Cookie flag that stops JavaScript reading the cookie, enforced by browsers
SameSite=Lax
Cookie flag that limits cross-site sending, enforced by browsers
Claim table
Claim, test, result for each statement you rely on
verify-prompt.md
Template asking for assumptions, numbered claims, confidence and tests

Before the next day

Mission: Claim, test, result table

Ask both assistants the same question about a header or cookie flag, then test what they say against your local server. The point is to practise treating answers as claims, and to see where the two assistants differ. Everything runs on your own machine.

About 35 min

You are done when: claims-table.md with at least five tested claims from two assistants, plus verify-prompt.md in your workbench folder.

Want more? Find one claim neither assistant could be tested on locally and write down what extra setup, such as HTTPS with a self-signed certificate, would let you test it.

Quiz

4 questions. Your score is saved with your progress.

1. An assistant says a header is 'Secure-Content-Policy' and gives its default value. What is the best first move?

2. Why does the template ask for a test command with each claim?

3. Both assistants agree that a cookie flag behaves a certain way. What does that tell you?

4. In the fake tests lab, why is 100 percent coverage not good evidence?

Finish the day

Mark the day complete to save your progress.

Go to Day 4

Next: Day 4 is a module review. You will take a scope statement and build a verified, AI-assisted recon plan, using today's claim table habit and Day 2's redaction rules together.

Tell someone you are doing this:WhatsAppXLinkedIn

Your notes

Private to this device. Write what you want to remember from today.