Day 11 of 1260 minCipher CadetHands-on lab
Day 11: Misinformation, Unbounded Use and Human Oversight
Taught by Mr Elite Updated
By the end of today you can
- Spot AI output that sounds sure but is wrong, including tests that prove nothing
- Explain how unlimited use drives up cost and enables denial of service
- Set simple limits such as rate caps, token caps and budgets for an LLM feature
- Decide which agent actions need human approval and which can run alone
Warm-up
Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.
Name the three roots of excessive agency from yesterday.
Too much functionality, too many permissions and too much autonomy.
Where should tool rules be enforced, and why?
In code outside the model, because a prompt rule can be talked around by an injection.
A developer asks an AI assistant to write tests for a discount function. Seven tests appear, the coverage report says 100 percent, and every tick is green. The pull request is approved in two minutes. Three weeks later a customer gets 100 percent off an order, and nobody can work out how the tests missed it.
Today is about trust and limits. Models say wrong things with total confidence. Applications that let anyone call a model without limits can be drained of money or knocked over. And agents that act without asking can do damage fast. You will finish by deciding where a human must say yes.
Confident and wrong: misinformation and overreliance
A language model predicts likely text. It does not look facts up and check them. When it does not know, it still produces something that reads fluently. That fluent wrong answer is called a hallucination, and OWASP lists the wider problem as misinformation.
The second half of the risk is overreliance. That is when people stop checking because the output looks polished. A tidy answer feels trustworthy, and that feeling is exactly what an attacker or a plain mistake can exploit.
- Hallucination
- A made-up answer that sounds real, such as an invented function name or a fake legal case.
- Overreliance
- Trusting AI output without checking it, because it looks confident.
- False assurance
- A signal that says everything is fine when nothing has really been verified.
The habit that protects you is simple: treat AI output as a draft from a keen junior colleague who never admits doubt. Anything that matters, such as a package name, a command, a legal claim or a security setting, gets checked against a source you trust before you act on it.
Here is a safe way to check a package name an assistant gave you, without installing anything. This runs on your own machine and only asks the public package index whether the name exists.
pip index versions requestsFor a real package you should see a line starting with the name and its newest version, then a list under Available versions. For a name that does not exist you will see an error saying no matching distribution was found. If you get that error, do not assume it is a typo and try variations. Ask where the name came from, and check the project's official documentation instead.
Quick check
Why does a hallucinated answer often go unnoticed?
Fluent, confident wording hides the fact that nothing was checked.
False assurance: tests that test nothing
Tests are meant to fail when code is wrong. An AI assistant can write tests that run the code, raise the coverage number, and still check nothing. Coverage only says a line was executed. It does not say the result was verified.
Here is a weak test and a real one for a function discount(price, percent). Read them side by side.
# Weak: passes even if discount() is badly broken
def test_discount_runs():
result = discount(100, 10)
assert result is not None
# Real: pins down the exact expected answer
def test_discount_ten_percent():
assert discount(100, 10) == 90Other warning signs are tests with no assert at all, asserts that compare a value to itself, tests that mock the very function they claim to test, and tests that only cover the happy path with no zero, negative or over-100 values.
That last point is how the opening story happened. The function had no guard for a percent above 100, and every generated test used a friendly number like 10 or 20. Coverage hit 100 percent because every line ran. Nobody asked what happens at the edges, so nobody found the bug.
Quick check
A test asserts that the result of discount(100, 10) is not None. What is wrong with it?
It checks that a value exists, not that the value is correct.
Unbounded consumption: cost and denial of service
Every model call costs money and computing power. Providers charge by tokens, which are small chunks of text, both in and out. If your app lets anyone send unlimited or enormous requests, two things can happen.
- Denial of service (DoS)
- Making a service slow or unavailable by overloading it.
- Denial of wallet
- Running up someone's cloud bill with huge or endless requests until the money, not the server, runs out.
- Rate limit
- A cap on how many requests one user or key can make in a period.
- Token cap
- A maximum on input or output length for one request.
Let me put numbers on it. These are made-up prices, only to show the shape of the problem. Say a provider charges £2 for every million tokens. One request carrying a 20,000 token document costs about 4 pence. A script sending 1,000 of them uses 20 million tokens and costs about £40, in a few minutes, from one person.
Agents make this worse. An agent stuck in a loop, or told by an injection to keep researching forever, can make hundreds of calls with nobody watching. Defences are plain: rate limits per user, token caps per request, a timeout, a loop or step limit for agents, a spending budget with alerts, and login before use.
per_user_rate_limit : 20 requests / minute
max_input_tokens : 4000 per request
max_output_tokens : 800 per request
request_timeout : 30 seconds
agent_max_steps : 10 per task
monthly_budget : 50 GBP, alert at 50 percent and 80 percent
login_required : yesHuman approval for risky actions
Yesterday's rule was that send, delete, pay and run code need approval. Today you refine it, because gating everything makes an agent useless and people start clicking Approve without reading. That habit is called approval fatigue.
Ask three questions of each action. Can it be undone? Does it leave the organisation or touch money, access or other people? How bad is the worst case if the agent was hijacked? Reading a calendar is low risk. Emailing an outside address, deleting files or paying an invoice is high risk.
Action Reversible? Leaves org/money? Needs human?
Read own calendar yes no no
Draft an email yes no no
Send email outside no yes yes
Delete a file no no yesNotice the rule of thumb the table follows. If an action fails any one of the three questions, it goes to a human. If it passes all three, the agent can run it alone, and you log what it did so you can review it later.
The approval screen matters as much as the rule. Compare a vague prompt with a useful one.
Vague: Allow action? [Yes] [No]
Useful: Send email to accounts@outside-example.com
Subject: Updated bank details
Body: (full text shown here)
Triggered by: a document you asked me to summarise
[Send] [Edit] [Cancel]Quick check
Why not require approval for every single agent action?
Too many prompts train people to click through, which defeats the control.
Watch me do it: audit one AI-written test
Before the labs, I will show you the method on your own machine. You need only a text editor. No Python install is required, because we are reading, not running. Nothing here touches any other system.
def test_a():
assert discount(200, 50) == 100
def test_b():
assert discount(200, 50) is not None
def test_c():
x = discount(200, 50)
assert x == xWhat just happened is that you judged tests by whether they could fail, not by how they looked. That one question works on any AI-written test, and it takes under a minute per test once you are used to it.
Your turn: the two labs
Now you apply the same thinking to a larger set. The first lab, ai-dev-fake-tests-1, is a safe practice environment built by SecurityElites. It holds seven AI-written tests for a discount function that report 100 percent coverage. Everything you do stays inside the lab.
Your turn
Tests That Test Nothing
This lab gives you seven AI-written tests for a discount function with 100 percent coverage. Your job is to work out which ones give false assurance, using the break question from the worked example. A second lab follows, on approvals, and I want you to do both today.
What to try
What to notice
- Coverage was 100 percent, yet several tests would survive a broken function.
- The weak tests look professional, which is why a quick review misses them.
- In the approvals lab, gating everything is as much a failure as gating nothing.
If it did not work
Treating a coverage percentage as proof the code is correct
Read the asserts. Ask whether each test would fail if the code were broken.
Adding only a request-count rate limit
Also cap input size, output tokens, agent steps and total spend, with an alert.
Requiring approval for every action
Gate only actions that are irreversible, external, or involve money or access.
Showing a vague approval prompt
Show the exact recipient, content or amount so the human can make a real decision.
Recap
- Models can be confidently wrong, and overreliance turns that into real harm
- Coverage measures execution, not correctness, so read the asserts
- Empty tests give false assurance and are worse than no tests because people trust them
- Unbounded use creates denial of service and denial of wallet risk
- Cap rate, size, steps and spend, and alert on spikes
- Gate risky actions by reversibility, reach and worst case, and keep the prompts few and clear
Cheat sheet
Break question- If the code were broken, would this test fail?
Hallucination- Fluent but invented output
Denial of wallet- Abuse that drains your budget
Rate limit + token cap- Limit how often and how big each request is
Step limit- Maximum number of actions an agent may take per task
Approval test- Reversible? External or money? Worst case?
Approval fatigue- People click Approve without reading when asked too often
pip index versions NAME- Checks that a package exists before you trust a name an AI gave you
Before the next day
Mission: Two labs and an approvals note
Finish both labs, then write a short note that shows you can judge where a human must step in. This turns the ideas from this lesson into a habit you can use on real projects.
About 35 min
You are done when: day11-approvals-note.txt plus two lab screenshots.
Want more? Rewrite one of the empty tests from the lab into a real test with an exact expected value and an edge case.
Quiz
4 questions. Your score is saved with your progress.
1. A test suite shows 100 percent coverage. What can you safely conclude?
Coverage records execution only. The asserts decide whether anything is actually checked.
2. An attacker sends thousands of huge documents to your chatbot and your cloud bill soars. Which risk is this?
Unlimited model use lets abuse turn directly into cost and outages.
3. Which action most clearly needs human approval before an agent runs it?
Payments are irreversible, involve money and have a severe worst case if the agent was hijacked.
4. What is the main danger of asking for approval on every action?
Constant prompts teach people to click through, so the control stops protecting anything.
Finish the day
Mark the day complete to save your progress.
Next: Day 12 is the capstone. You will audit a small AI application against all ten OWASP LLM risks and rank your findings with fixes, using everything from injection to the limits and approvals you met today.
Your notes
Private to this device. Write what you want to remember from today.