Day 11 of 1260 minCipher CadetHands-on lab

Day 11: Misinformation, Unbounded Use and Human Oversight

Taught by Mr Elite Updated

By the end of today you can

  • Spot AI output that sounds sure but is wrong, including tests that prove nothing
  • Explain how unlimited use drives up cost and enables denial of service
  • Set simple limits such as rate caps, token caps and budgets for an LLM feature
  • Decide which agent actions need human approval and which can run alone

Warm-up

Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.

Name the three roots of excessive agency from yesterday.

Too much functionality, too many permissions and too much autonomy.

Where should tool rules be enforced, and why?

In code outside the model, because a prompt rule can be talked around by an injection.

A developer asks an AI assistant to write tests for a discount function. Seven tests appear, the coverage report says 100 percent, and every tick is green. The pull request is approved in two minutes. Three weeks later a customer gets 100 percent off an order, and nobody can work out how the tests missed it.

Today is about trust and limits. Models say wrong things with total confidence. Applications that let anyone call a model without limits can be drained of money or knocked over. And agents that act without asking can do damage fast. You will finish by deciding where a human must say yes.

Confident and wrong: misinformation and overreliance

A language model predicts likely text. It does not look facts up and check them. When it does not know, it still produces something that reads fluently. That fluent wrong answer is called a hallucination, and OWASP lists the wider problem as misinformation.

The second half of the risk is overreliance. That is when people stop checking because the output looks polished. A tidy answer feels trustworthy, and that feeling is exactly what an attacker or a plain mistake can exploit.

Hallucination
A made-up answer that sounds real, such as an invented function name or a fake legal case.
Overreliance
Trusting AI output without checking it, because it looks confident.
False assurance
A signal that says everything is fine when nothing has really been verified.

The habit that protects you is simple: treat AI output as a draft from a keen junior colleague who never admits doubt. Anything that matters, such as a package name, a command, a legal claim or a security setting, gets checked against a source you trust before you act on it.

Here is a safe way to check a package name an assistant gave you, without installing anything. This runs on your own machine and only asks the public package index whether the name exists.

bash
pip index versions requests
Asks the package index for the versions of a package. Replace requests with the name you want to check. Needs pip 21.2 or newer.

For a real package you should see a line starting with the name and its newest version, then a list under Available versions. For a name that does not exist you will see an error saying no matching distribution was found. If you get that error, do not assume it is a typo and try variations. Ask where the name came from, and check the project's official documentation instead.

Quick check

Why does a hallucinated answer often go unnoticed?

False assurance: tests that test nothing

Tests are meant to fail when code is wrong. An AI assistant can write tests that run the code, raise the coverage number, and still check nothing. Coverage only says a line was executed. It does not say the result was verified.

Here is a weak test and a real one for a function discount(price, percent). Read them side by side.

python
# Weak: passes even if discount() is badly broken
def test_discount_runs():
    result = discount(100, 10)
    assert result is not None

# Real: pins down the exact expected answer
def test_discount_ten_percent():
    assert discount(100, 10) == 90
The first test only checks that something came back. Return 0, 100 or 5000 and it still passes. The second fails on any wrong value.

Other warning signs are tests with no assert at all, asserts that compare a value to itself, tests that mock the very function they claim to test, and tests that only cover the happy path with no zero, negative or over-100 values.

That last point is how the opening story happened. The function had no guard for a percent above 100, and every generated test used a friendly number like 10 or 20. Coverage hit 100 percent because every line ran. Nobody asked what happens at the edges, so nobody found the bug.

Quick check

A test asserts that the result of discount(100, 10) is not None. What is wrong with it?

Unbounded consumption: cost and denial of service

Every model call costs money and computing power. Providers charge by tokens, which are small chunks of text, both in and out. If your app lets anyone send unlimited or enormous requests, two things can happen.

Denial of service (DoS)
Making a service slow or unavailable by overloading it.
Denial of wallet
Running up someone's cloud bill with huge or endless requests until the money, not the server, runs out.
Rate limit
A cap on how many requests one user or key can make in a period.
Token cap
A maximum on input or output length for one request.

Let me put numbers on it. These are made-up prices, only to show the shape of the problem. Say a provider charges £2 for every million tokens. One request carrying a 20,000 token document costs about 4 pence. A script sending 1,000 of them uses 20 million tokens and costs about £40, in a few minutes, from one person.

Agents make this worse. An agent stuck in a loop, or told by an injection to keep researching forever, can make hundreds of calls with nobody watching. Defences are plain: rate limits per user, token caps per request, a timeout, a loop or step limit for agents, a spending budget with alerts, and login before use.

text
per_user_rate_limit : 20 requests / minute
max_input_tokens    : 4000 per request
max_output_tokens   : 800 per request
request_timeout     : 30 seconds
agent_max_steps     : 10 per task
monthly_budget      : 50 GBP, alert at 50 percent and 80 percent
login_required      : yes
A sample set of limits for a small AI feature. The numbers are examples. The point is that every dimension has a ceiling.

Human approval for risky actions

Yesterday's rule was that send, delete, pay and run code need approval. Today you refine it, because gating everything makes an agent useless and people start clicking Approve without reading. That habit is called approval fatigue.

Ask three questions of each action. Can it be undone? Does it leave the organisation or touch money, access or other people? How bad is the worst case if the agent was hijacked? Reading a calendar is low risk. Emailing an outside address, deleting files or paying an invoice is high risk.

text
Action              Reversible?  Leaves org/money?  Needs human?
Read own calendar   yes          no                 no
Draft an email      yes          no                 no
Send email outside  no           yes                yes
Delete a file       no           no                 yes
A simple decision table. Use the same three columns in the note you write for the mission.

Notice the rule of thumb the table follows. If an action fails any one of the three questions, it goes to a human. If it passes all three, the agent can run it alone, and you log what it did so you can review it later.

The approval screen matters as much as the rule. Compare a vague prompt with a useful one.

text
Vague:  Allow action? [Yes] [No]

Useful: Send email to accounts@outside-example.com
        Subject: Updated bank details
        Body: (full text shown here)
        Triggered by: a document you asked me to summarise
        [Send] [Edit] [Cancel]
The useful version shows the recipient, the content and what triggered it, so a human can spot something odd.

Quick check

Why not require approval for every single agent action?

Watch me do it: audit one AI-written test

Before the labs, I will show you the method on your own machine. You need only a text editor. No Python install is required, because we are reading, not running. Nothing here touches any other system.

python
def test_a():
    assert discount(200, 50) == 100

def test_b():
    assert discount(200, 50) is not None

def test_c():
    x = discount(200, 50)
    assert x == x
Three tests, one useful. Test C is always true, whatever the function returns.

What just happened is that you judged tests by whether they could fail, not by how they looked. That one question works on any AI-written test, and it takes under a minute per test once you are used to it.

Your turn: the two labs

Now you apply the same thinking to a larger set. The first lab, ai-dev-fake-tests-1, is a safe practice environment built by SecurityElites. It holds seven AI-written tests for a discount function that report 100 percent coverage. Everything you do stays inside the lab.

Your turn

Tests That Test Nothing

This lab gives you seven AI-written tests for a discount function with 100 percent coverage. Your job is to work out which ones give false assurance, using the break question from the worked example. A second lab follows, on approvals, and I want you to do both today.

Open the lab in a new tab

What to try

What to notice

  • Coverage was 100 percent, yet several tests would survive a broken function.
  • The weak tests look professional, which is why a quick review misses them.
  • In the approvals lab, gating everything is as much a failure as gating nothing.

If it did not work

Treating a coverage percentage as proof the code is correct

Read the asserts. Ask whether each test would fail if the code were broken.

Adding only a request-count rate limit

Also cap input size, output tokens, agent steps and total spend, with an alert.

Requiring approval for every action

Gate only actions that are irreversible, external, or involve money or access.

Showing a vague approval prompt

Show the exact recipient, content or amount so the human can make a real decision.

Recap

  • Models can be confidently wrong, and overreliance turns that into real harm
  • Coverage measures execution, not correctness, so read the asserts
  • Empty tests give false assurance and are worse than no tests because people trust them
  • Unbounded use creates denial of service and denial of wallet risk
  • Cap rate, size, steps and spend, and alert on spikes
  • Gate risky actions by reversibility, reach and worst case, and keep the prompts few and clear

Cheat sheet

Break question
If the code were broken, would this test fail?
Hallucination
Fluent but invented output
Denial of wallet
Abuse that drains your budget
Rate limit + token cap
Limit how often and how big each request is
Step limit
Maximum number of actions an agent may take per task
Approval test
Reversible? External or money? Worst case?
Approval fatigue
People click Approve without reading when asked too often
pip index versions NAME
Checks that a package exists before you trust a name an AI gave you

Before the next day

Mission: Two labs and an approvals note

Finish both labs, then write a short note that shows you can judge where a human must step in. This turns the ideas from this lesson into a habit you can use on real projects.

About 35 min

You are done when: day11-approvals-note.txt plus two lab screenshots.

Want more? Rewrite one of the empty tests from the lab into a real test with an exact expected value and an edge case.

Quiz

4 questions. Your score is saved with your progress.

1. A test suite shows 100 percent coverage. What can you safely conclude?

2. An attacker sends thousands of huge documents to your chatbot and your cloud bill soars. Which risk is this?

3. Which action most clearly needs human approval before an agent runs it?

4. What is the main danger of asking for approval on every action?

Finish the day

Mark the day complete to save your progress.

Go to Day 12

Next: Day 12 is the capstone. You will audit a small AI application against all ten OWASP LLM risks and rank your findings with fixes, using everything from injection to the limits and approvals you met today.

Tell someone you are doing this:WhatsAppXLinkedIn

Your notes

Private to this device. Write what you want to remember from today.