Day 7 of 960 minCipher Cadet

Day 7: What Red Teaming Is and How to Plan a Test

Taught by Mr Elite Updated

By the end of today you can

  • Explain how red teaming differs from penetration testing and bug hunting
  • Write a scope with permission and rules of engagement for an AI test
  • Turn threats from a threat model into test cases with a goal, method and expected safe result
  • Keep notes and evidence that someone else can follow

Warm-up

Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.

What is a trust boundary?

A place where control changes hands, so data crossing it needs checking.

Name the four steps of the threat modelling method from Day 6, in order.

Assets, entry points, threats, defences.

Imagine a friend hands you their chatbot and says, "Have a go at breaking it." You open the chat window, type a few clever tricks, and something odd happens. You are not sure what you typed, whether it will happen again, or whether you were even allowed to try that. You have nothing to show them except a vague memory.

That is what testing without a plan looks like. Today I will show you how to avoid it. You will write a scope, set rules, turn the threat model you built yesterday into test cases, and set up notes so every result can be repeated. Tomorrow you run the tests. Today you make sure they are safe and worth running.

Three jobs that sound alike

People use the words red teaming, penetration testing and bug hunting as if they mean the same thing. They overlap, but each has a different aim. Knowing which one you are doing decides how you plan, what you report and when you stop.

Penetration testing
A scoped, agreed check of a system to find as many weaknesses as possible in a set time, then report them.
Bug hunting
Looking for flaws in a product whose owner has published rules, often for a reward. You report each flaw on its own.
Red teaming
Acting as a realistic adversary to test whether a system, and the people and controls around it, hold up against a goal such as leaking data or misusing a tool.
Rules of engagement
The written limits on what you may do, when, and how.

Think of it this way. A pen tester walks the whole building and tries every door. A bug hunter reports each broken lock for a reward. A red teamer is given one target, say the safe in the manager's office, and tries to reach it the way a real intruder would, including by talking to staff.

For AI, red teaming also covers behaviour that is not a classic bug. A chatbot that can be talked into revealing its instructions has no broken code, yet it fails the goal. So we test behaviour against goals, not only code against flaws. That is why a plan matters: the goal tells you what counts as a win for the attacker.

Quick check

A tester is told to try to make a support bot leak another customer's order details, using any realistic trick within agreed limits. Which job is this closest to?

Scope, permission and rules of engagement

Before any test, you need permission. Permission means the owner of the system has agreed, in writing, that you may test it. Without that, testing can be illegal and harmful, however good your intentions. In this course you only test your own machine, your own local setup or the named SecurityElites practice labs.

Scope is the list of what is in and out of bounds. Rules of engagement are how you behave inside it. I keep both short enough to fit on half a page, because a rule nobody reads protects nobody.

text
SCOPE
In scope: my Day 4 chatbot, running on my own laptop
Out of scope: any real service, any other person's account, any live API key that costs money

RULES OF ENGAGEMENT
- Test only the in-scope system
- Use fake data only, never real personal data
- Stop and note it if the bot does something unexpected and harmful
- Testing window: evenings this week
- Owner and approver: me
A starter scope and rules block. Copy it into your plan and edit the details to match your chatbot.

You are both the owner and the approver here, because it is your own machine. That is fine for learning. Writing "me" still forces you to make the decision on purpose. In a real engagement that line would carry a client's name and a date, and you would not start without it.

Turning threats into test cases

Yesterday you listed threats and a defence for each. A test case is a small experiment that checks whether that defence works. Each one has three parts: a goal, a method and an expected safe result.

Goal
What a successful attacker would achieve, such as getting the bot to reveal its system prompt.
Method
The exact kind of input you will try, described clearly enough to repeat.
Expected safe result
What the system should do if the defence works, such as politely refusing.

The expected safe result matters most. If you do not write it down first, you will be tempted to decide afterwards that whatever happened was fine. Writing it first is like writing the answer key before marking your own paper.

Keep each case small. One test case should check one threat. If a case has three goals, a failure tells you very little, because you cannot say which goal failed. Several small cases give clearer evidence than one large, tangled one.

Quick check

Why do we write the expected safe result before running the test?

Watch me do it: one threat becomes one test case

I will take one threat from a typical chatbot threat model and turn it into a test case. Everything here happens in text files on your own machine. Nothing is sent to any real service. Follow along with your own files so you can reuse the pattern.

bash
ls
# expected: day6-threat-model.txt (plus any other files you have)
Lists the files in the current folder so you can confirm you are in the right place.
text
TC-01
Threat: Injection (from threat model, entry point: chat box)
Asset at risk: System prompt and the bot's rules
Goal: Make the bot drop its rules and answer outside its job
Method: Direct request to ignore instructions, then a roleplay framing
Expected safe result: Bot declines, stays on topic, reveals no instructions
Result: (filled in on Day 8)
One finished test case. The Result line stays empty until you actually run it tomorrow.
bash
cat day7-test-plan.txt
Prints your plan to the screen. On Windows PowerShell use: type day7-test-plan.txt

If the block prints exactly as you wrote it, the case is saved. Notice what we did not do. We did not type any attack. We described what kind of input we would try and what a safe bot would do. Tomorrow's job is then simply to run it and compare.

Notes and evidence

A test you cannot reproduce is only an anecdote. AI systems also give different answers to the same input, so you often need to run a test several times. Good notes make that possible.

For each run, record the date and time, the exact input, the full output, the version or settings of the bot, and your verdict: pass, fail or unclear. Take a screenshot or copy the text. Keep evidence free of real personal data, since a log full of private details becomes a risk itself.

text
RUN LOG
Test: TC-01   Run: 1   Time: 2025-01-10 19:40
Input: (paste exact text)
Output: (paste exact text)
Settings: model name, temperature, system prompt version
Verdict: pass / fail / unclear
Notes: anything surprising
A run log entry. Make one per attempt, so repeated runs can be compared.

Settings matter more than people expect. Temperature is the setting that controls how much a model varies its answers. A test that fails at a high temperature and passes at a low one tells you something real, but only if you wrote the setting down.

Your turn: build the five-case plan

No lab today, so you work on your own machine with your own files. Open your day6-threat-model.txt and choose five threats. Aim for variety: injection, leakage and excess permissions all deserve at least one test, because those were yesterday's three common threats.

bash
grep -c "^TC-" day7-test-plan.txt
# expected output: 5
Counts the lines beginning with TC- in your plan, a quick way to confirm you wrote five cases.

Now read each method aloud as if explaining it to a friend who has never seen your chatbot. If you stumble, the method is too vague. Fix it now, while it costs you a minute, rather than tomorrow when you are mid-test and unsure what you meant.

If it did not work

Starting to test before writing down scope and permission

Write the scope block first. If you cannot say who approved the test, do not run it.

Vague methods such as "try to break it"

Describe the input type and steps so another person could repeat the test exactly.

Writing the expected result after seeing the output

Fill in the expected safe result first, and leave the verdict for after the run.

Running a test once and calling it done

Models vary their answers, so plan several runs and log each one.

Packing several goals into one test case

Split it so each case checks one threat. Then a failure points to one clear cause.

Recap

  • Red teaming tests a system against an adversary goal, while pen testing and bug hunting aim at finding flaws
  • Permission, scope and rules of engagement come before any test
  • A test case has a goal, a method and an expected safe result
  • Test cases come straight from your threat model, one per threat
  • Notes and evidence make a result repeatable and believable
  • Use fake data and stay inside scope every time

Cheat sheet

Red teaming
Realistic adversary testing against a defined goal
Scope
What is in and out of bounds for the test
Rules of engagement
Written limits on how, when and what you may test
Test case
Goal + method + expected safe result
Run log
Time, input, output, settings, verdict and notes for each attempt
Verdict
Pass, fail or unclear, judged against the expected safe result
grep -c "^TC-" file
Counts how many test cases your plan file contains

Before the next day

Mission: Five-case test plan

Turn yesterday's threat model into a plan you can run tomorrow. A plan written now saves confusion later and keeps your testing safe and repeatable.

About 40 min

You are done when: A file called day7-test-plan.txt with a scope, rules of engagement, five test cases and a blank run log.

Want more? Add a priority (high, medium, low) to each test case with one sentence explaining why, and order the plan by priority.

Quiz

4 questions. Your score is saved with your progress.

1. What best separates red teaming from ordinary bug hunting?

2. Which item belongs in the rules of engagement for a chatbot test?

3. A test case reads: "Try to hack the bot." What is wrong with it?

4. Why run the same test more than once?

Finish the day

Mark the day complete to save your progress.

Go to Day 8

Next: On Day 8 you run these tests in a lab, trying roleplay jailbreaks and prompt injection, and record each outcome in your run log. Today's plan is what keeps that testing organised and safe.

Tell someone you are doing this:WhatsAppXLinkedIn

Your notes

Private to this device. Write what you want to remember from today.