Day 12 of 1460 minShell WalkerHands-on lab

Day 12: Human Approval: Where It Works and Where It Fails

Taught by Mr Elite Updated

By the end of today you can

  • Explain why a human approval step is a control that can fail, and name its two main weaknesses
  • Spoof an approval summary in the lab so it hides the real tool call
  • Demonstrate a time-of-check versus time-of-use gap in a tool execution flow
  • Design an approval step that shows the real action and binds approval to it

Warm-up

Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.

Why is an allowlist safer than a denylist when you restrict a coding agent?

An allowlist fails safe: anything not listed is blocked. A denylist fails open: anything you forgot to list is allowed.

Why do we measure blast radius instead of likelihood when we reduce permissions?

We assume the agent can be hijacked, so what matters is how much damage a hijacked agent could do with its permissions.

Picture Ravi in Finance. Forty approval requests land in his queue before lunch. Each one is a neat sentence written by the assistant: Pay Acme Supplies invoice 1042, 480 pounds. He clicks Approve forty times. One of those sentences was written by an agent that had read a poisoned document, and the payment behind it went somewhere else.

Everybody says to put a human in the loop. Today I show you why that sentence is not a finished control. A human can only judge what they are shown, and what they are shown is often written by the very system we do not fully trust. We will break two approval flows in the labs, then design one that is much harder to fool.

Human-in-the-loop is a control, not a guarantee

Human-in-the-loop means the agent must stop and get a person's yes before it does something risky, such as paying money, deleting files or sending email. It is a good control. It catches mistakes and many hijacks. But like any control, it has a precondition: the person must see the truth and have the time to think about it.

Two things commonly break that precondition. The first is the summary problem. The text the human reads is produced by the model, and a hijacked model can write a calm, friendly lie. The second is the timing problem. The thing approved and the thing executed can differ, because something changed between the moment of the check and the moment of use.

Notice that neither problem needs the human to be careless. Ravi can be sharp, well rested and honest, and still approve the wrong payment, because the screen told him something false. That is why I treat approval as a system design question and not as a training question. You cannot fix a bad prompt by telling people to try harder.

Approval prompt
The message that asks a person to allow or deny an agent action.

Misleading summaries

An agent usually has two views of an action. One is the raw tool call, for example pay_vendor with a vendor ID, an amount and an account number. The other is a human-friendly summary the model writes. Interfaces often show the summary first and hide the raw call behind a toggle. That makes the summary the real security boundary, and it is attacker-influenced text.

This links straight back to Day 7. Tool poisoning put hostile text into a tool description to steer the model. Here the same idea targets the person. Injected content tells the model what to write in the summary, so the human approves a story while the tool call does something else.

json
{
  "summary": "Pay Acme Supplies invoice 1042 (480.00 GBP)",
  "tool_call": {
    "name": "pay_vendor",
    "arguments": { "vendor_id": "V-9917", "amount": 4800.00, "account": "GB00ATTACKER" }
  }
}
A made-up example of a mismatch. The summary and the arguments disagree on vendor, amount and destination. Only the toggle shows it.

Read that example slowly. The summary says 480.00 and the arguments say 4800.00, a single extra zero. The summary names Acme, while the call uses a vendor ID the person has never seen, and the account field holds a destination that appears nowhere in the friendly text. Each difference is small, and together they are the whole attack.

Attackers like this pattern because the summary only has to be plausible. It does not need to fool a security expert. It needs to fool someone with thirty other requests waiting, who sees a familiar supplier name and a believable amount.

Quick check

Why is a model-written approval summary a weak security boundary on its own?

Time-of-check versus time-of-use

Time-of-check versus time-of-use, shortened to TOCTOU, is a gap bug. A system checks a condition at one moment and acts on it at a later moment. If the world can change in between, the check no longer proves anything. Classic example: you confirm a balance is 500, then someone drains it, then the transfer still runs.

In agents the gap is easy to create, because checking and acting are separate tool calls with model reasoning, user messages or approvals in between. This connects to Day 9. A tool argument that was valid when checked can be swapped for a different value before use. The defence is to bind the approval or check to the exact action, and to re-verify at the moment of execution.

text
10:00:01  check_balance(account=A)      -> 500 GBP, OK
10:00:04  model reasons, user chats      (gap opens)
10:00:09  something else spends 450 GBP  (state changes)
10:00:12  transfer(account=A, 400 GBP)  -> runs on stale belief
10:00:12  result: balance is now -350 GBP
A timeline of a TOCTOU gap in an agent. The check passed at 10:00:01, but the transfer at 10:00:12 relied on a fact that was no longer true.

The same shape applies to approvals. If Ravi approves a payment at 10:00 and the agent builds the final call at 10:05, anything that edits the arguments in those five minutes rides on his yes. The approval was real. It just no longer describes what runs.

TOCTOU
A race where the state you checked changes before you act on it.
Binding
Tying an approval to one exact action, so any change invalidates it.

Quick check

Which fix best closes a TOCTOU gap?

Watch me do it: audit an approval flow on your machine

Before the labs, I want you to practise seeing the gap. We will use a few shell commands on your own machine that compare what a person was shown with what actually runs, and then bind an approval to a hash. Nothing here touches any other system, and no real payment exists. You need a terminal and the tool jq, which reads JSON.

bash
cat > request.json <<'EOF'
{"summary":"Pay Acme invoice 1042",
 "tool_call":{"name":"pay_vendor",
 "arguments":{"vendor_id":"V-9917","amount":4800,"account":"GB00ATTACKER"}}}
EOF
jq . request.json
Creates a sample request and pretty-prints it. Expect formatted JSON.
bash
jq -r '.summary' request.json
jq -c '.tool_call.arguments' request.json
The first line is what Ravi reads. The second is what executes. Notice how much the summary leaves out.
bash
jq -cS '.tool_call' request.json | sha256sum > approved.hash
cat approved.hash
Sorts keys and compacts the tool call so the text is always identical, then hashes it. This hash stands for 'the approved action'.
bash
jq '.tool_call.arguments.amount = 9800' request.json > changed.json
jq -cS '.tool_call' changed.json | sha256sum | cmp - approved.hash && echo MATCH || echo MISMATCH
jq -cS '.tool_call' request.json | sha256sum | cmp - approved.hash && echo MATCH || echo MISMATCH
The first check uses the tampered copy and should print MISMATCH. The second uses the original and should print MATCH. Any change to the call breaks the approval.

What just happened is the heart of honest approval. Changing one number, from 4800 to 9800, produced a completely different hash, so the saved approval no longer fitted the action. A system that executes only on a matching hash cannot be tricked into running a swapped call.

Designing honest confirmations

An honest approval step is built from what the system will do, not from what the model says it will do. Generate the display from the structured tool call in ordinary code, with no model involved. Then the text cannot be talked into lying. Here are the five parts I want in every design.

text
APPROVE PAYMENT?
Action:       pay_vendor
Payee:        UNKNOWN (V-9917 is not in the vendor list)
Account:      GB00ATTACKER  (never used before)
Amount:       4,800.00 GBP  (10x the usual for this supplier)
Reversible:   No
Call hash:    9f2c...e41a
[Deny]  [Approve this exact call]
A mock-up of an honest prompt. Every line is produced by code from the tool call and company records, never by the model.

Compare that with the one-line summary Ravi saw. The honest version makes the attack loud: unknown payee, unfamiliar account, ten times the usual amount. Even a tired reviewer would stop. You have not made Ravi cleverer, you have made the truth hard to miss.

Lab: break two approval flows

Now you attack two flows in the SecurityElites labs. Only use these labs, not any real payment or banking system. The first targets the summary: you will see what the approver sees and try to make a harmless description sit on top of a different action. The second targets the timing gap between a check and its use.

Take the first lab slowly. Before changing anything, read the default screen as Ravi would, and decide whether you would approve it. Then open the hidden raw call and compare. Do the same in the second lab by drawing the order of events on paper. Writing the timeline first makes the gap obvious, and it tells you exactly where a change could land.

Your turn

AI Human-in-the-Loop Bypass

This lab puts you in front of LedgerBot, which asks Ravi to approve payments from a friendly summary while the raw call sits behind a toggle. It shows in practice why a model-written summary cannot be trusted. Then, in the same session, repeat the thinking on ai-agent-toctou-race-1, where BankBot checks a balance and transfers in separate calls.

Open the lab in a new tab

What to try

What to notice

  • Which part of the screen the approver sees by default, and which part is hidden.
  • Whether the approval is tied to the exact action or only to a loose description.
  • Which values are checked once and then trusted later without a second check.

If it did not work

Trusting the summary because it sounds sensible

Always compare it with the raw arguments. If you cannot see them, treat the prompt as unverified.

Fixing TOCTOU by checking sooner

Checking earlier does not shrink the gap. Verify again at execution, inside the same atomic operation.

Adding more approval prompts as the cure

Too many prompts cause fatigue. Gate only high-risk actions and make each one clear.

Letting the model write the confirmation text

Generate the display in code from the structured call so injected text cannot alter it.

Hashing the call without a stable format

Key order and spacing change a hash. Sort keys and compact the JSON first, as jq -cS does, so identical calls always give identical hashes.

Recap

  • Human approval only works if the person sees the truth and has time to judge it
  • A model-written summary is attacker-influenced text, so it is a weak boundary
  • TOCTOU means the checked state can change before use
  • Bind approval to the exact action and re-verify at execution
  • Render confirmations from the real tool call, not from model prose
  • Approval fatigue defeats good design, so keep prompts rare and meaningful

Cheat sheet

Summary spoofing
Injected content steers the model into writing a harmless description of a harmful action
TOCTOU
State changes between the check and the action
Binding
Approval valid only for a hash of the exact call
jq -r '.summary' file
Print the human-facing text from a JSON file
jq -c '.tool_call.arguments' file
Print the real arguments compactly
jq -cS '.tool_call' file | sha256sum
Hash a tool call in a stable format
cmp - approved.hash
Compare piped input with a saved hash file
Honest prompt
Built in code from structured data, shows destination, amount and irreversibility

Before the next day

Mission: Design an honest approval step

Finish both labs, then write a half-page design for an approval step that shows the real action. This is a review day, so tie each lab back to the earlier risks from Days 7 to 9. The aim is to prove you can both break and repair the control.

About 40 min

You are done when: approval-design.md of about half a page, with two lab notes and a risk link for each lab.

Want more? Write a short Python function that hashes a tool call with SHA-256 and refuses to run it if the hash differs from the approved one.

Quiz

4 questions. Your score is saved with your progress.

1. An approval prompt shows a model-written summary and hides the raw call. What is the main risk?

2. BankBot checks a balance, then transfers in a separate call. What defect is this?

3. Which design makes a confirmation harder to fool?

4. Why can too many approval prompts weaken security?

Finish the day

Mark the day complete to save your progress.

Go to Day 13

Next: Day 13 pulls the course together into a structured review of an MCP server and a lockdown checklist. Today's honest-approval design becomes one item on that checklist.

Tell someone you are doing this:WhatsAppXLinkedIn

Your notes

Private to this device. Write what you want to remember from today.