Day 1 of 1060 minShell WalkerHands-on lab
Day 1: Why AI Code Needs Review
Taught by Mr Elite Updated
By the end of today you can
- Explain why AI output must be treated as untrusted input
- Name the five risks this course covers: flaws, secrets, permissions, poisoned input and fake packages
- Confirm your terminal, git and editor are ready for the course
- Spot a test that passes even when the code is broken
Picture an assistant that writes a discount function and seven tests for it in under a minute. The coverage report says 100%. Every test is green. You merge it, and a week later customers are getting negative prices. Nothing about the process looked wrong, and that is exactly the problem.
This course is about that gap between code that looks finished and code that is safe. Over ten days I will show you how AI coding tools put your projects at risk, and give you repeatable checks to catch it. Today we set the frame, check your tools, and run your first lab.
What this course covers
AI coding assistants write code, suggest packages, read your files and, in agent mode, run commands for you. That is a lot of trust handed to a tool that produces confident text, not guaranteed correct text. I am not here to tell you to stop using them. I am here to show you how to use them and still ship code you can defend.
The course runs in three parts. Days 2 and 3 are about reviewing generated code. Days 4 to 8 cover secrets, packages, agent permissions, hostile input and tool connections. Days 9 and 10 turn it all into a team policy and a full audit of a project. Every hands-on step runs on your own machine or in the SecurityElites labs, never against anyone else's system.
Each day follows the same shape. I explain one idea, I show you a small example step by step, and then you practise it yourself. Plan on about an hour. You do not need to be an expert programmer, but you should be able to read a short function and follow what it does. If you can do that, you have enough to start.
- AI coding assistant
- A tool that suggests or writes code from your prompts, usually inside your editor or chat window.
- Agent mode
- A setting where the assistant does not only suggest code but also edits files and runs commands on your behalf.
Setup check: terminal, git, editor
Before the theory, let me make sure your tools work. This takes about five minutes and saves you trouble later. Everything below runs only on your own computer.
git --version
node --version && npm --version
python3 --version && pip --versiongit version 2.43.0
v20.11.0
10.2.4
Python 3.12.1
pip 24.0 from /usr/lib/python3/dist-packages/pip (python 3.12)AI output as untrusted input
Here is the single idea I want you to carry through all ten days. Treat everything an AI gives you the way you would treat data typed into a web form by a stranger. You would not run that data without checking it. The same rule applies to generated code, suggested packages and commands.
Why? A model predicts likely text. It does not know your threat model, your data or your deadline. It can copy a pattern that was common in its training material, including insecure patterns. It can sound certain while being wrong. And it can be steered by text it reads, which we will see on Day 7.
Notice what this does not say. It does not say the output is bad, and it does not say the assistant is hostile. Most of the time the code is fine. The point is that you cannot tell the good output from the bad by how it reads, because both read the same way. Only checking separates them.
- Untrusted input
- Any data or code you did not write and verify yourself. It might be fine, but you check it before relying on it.
- Threat model
- A short description of what you are protecting, who might attack it and how.
Quick check
A colleague says, 'The AI wrote it and the tests pass, so it is safe.' What is the best response?
Passing tests only show the code matches what the tests check, and both the code and the tests came from the same untrusted source.
The five risks
Everything in this course falls under five risks. Learn these names now. We will return to each one with a hands-on exercise.
- Insecure flaws
- Generated code that works but is vulnerable, such as building a database query by joining strings, which allows injection.
- Secrets
- API keys, passwords and tokens that leak when you paste them into a prompt or when the assistant writes them into code.
- Permissions
- An agent that can read files, run commands or reach the network has a blast radius, which is everything it could damage if it goes wrong or is hijacked.
- Poisoned input
- Hostile text in an issue, web page or document that an agent reads and then obeys as if it were your instruction.
- Fake packages
- Package names an AI invents. An attacker can register that name and wait for someone to install it. This is called slopsquatting.
Notice that today's lab does not fit neatly into one of those five. It sits underneath all of them: false confidence. Tests, green ticks and fluent explanations make us stop looking. Each of the five risks gets worse when we stop looking.
A quick example of how they stack. An assistant invents a package name, you install it, and the tests pass because nobody tested what the package does. One risk, hidden by another. Keep that picture in mind as we go, because real incidents are rarely tidy.
Quick check
An agent reads a GitHub issue containing hidden text telling it to send your environment variables elsewhere. Which risk is this?
The attack arrives as text the agent reads and then treats as an instruction, which is poisoned input.
Watch me do it: spotting a test that tests nothing
Before you open the lab, let me show you the idea on a tiny example you can type yourself. We will use Python, but the logic is the same in any language. Everything runs in a throwaway folder on your own machine, and you can delete it afterwards.
mkdir demo && cd demo# discount.py
def apply_discount(price, percent):
return price # bug: percent is ignored# test_discount.py
import unittest
from discount import apply_discount
class T(unittest.TestCase):
def test_returns_number(self):
self.assertIsInstance(apply_discount(100, 10), (int, float))
# honest version to swap in:
# def test_ten_percent(self):
# self.assertEqual(apply_discount(100, 10), 90)python3 -m unittest -vtest_returns_number (test_discount.T) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.000s
OKtest_returns_number (test_discount.T) ... ok
test_ten_percent (test_discount.T) ... FAIL
AssertionError: 100 != 90
Ran 2 tests in 0.001s
FAILED (failures=1)Look at what happened. The first test never said anything about the discount, so it could not notice the bug. The second test states a value I worked out myself, not one copied from the function. That independence is what gives a test its power. Fixing the function to return price * (100 - percent) / 100 would turn the second test green for the right reason.
Your lab: seven tests and 100% coverage
Now you do the same detective work at larger scale. The lab ai-dev-fake-tests-1 gives you a discount function and seven AI-written tests. Coverage reads 100%, yet some of those tests would pass on broken code. You are not asked to fix anything. You are asked to judge which tests deserve trust.
Your turn
Tests That Test Nothing
Now you try it at scale. In this lab an assistant wrote seven tests for a discount function and coverage reads 100%, but some tests would pass even if the function were completely broken. Your job is to find them before anyone trusts them. It is a beginner lab and runs entirely in your browser.
What to try
What to notice
- Coverage measures which lines ran, not whether anything was checked
- Every test looks tidy and well named, which makes the weak ones easy to miss
- The tests that check exact, independently known values are the ones that give real protection
If it did not work
Trusting a coverage number or a wall of green ticks
Coverage only proves lines ran. Read the assertions and ask whether each one would fail if the code were wrong.
Asking the same AI to confirm its own work is safe
Use the AI to draft, then review with a checklist you control. You will build one on Day 2.
Skipping setup checks and finding problems on Day 5
Run the version commands today. Fix missing tools now while nothing else depends on them.
Treating the AI as either useless or perfect
Treat it as a fast, fallible junior contributor. Useful, but every output gets reviewed.
Naming a test file without the test_ prefix, then seeing Ran 0 tests
Rename the file to start with test_ and run the command again from the same folder.
Recap
- AI output is untrusted input and needs review before it ships
- The five risks are insecure flaws, secrets, permissions, poisoned input and fake packages
- False confidence from tests and coverage makes every other risk harder to see
- A good test fails when the code is wrong; ask whether it would
- Your terminal, git and editor must work before we go further
Cheat sheet
git --version- Confirms git is installed
node --version / python3 --version- Confirms your runtime is installed
python3 -m unittest -v- Runs the tests in the current folder and lists each result
Untrusted input- Anything you did not write and verify yourself, including AI output
Blast radius- Everything a tool or agent could damage if it fails or is hijacked
Slopsquatting- Registering a package name an AI tends to invent, so installs pull in attacker code
Break-it check- Would this test fail if I broke the code on purpose?
Before the next day
Mission: Why the tests gave false confidence
Finish the lab, then put what you learned into your own words. Writing it down forces you to understand it and gives you the first entry in the notes file you will keep for the whole course.
About 30 min
You are done when: A file called ai-review-notes.md containing three sentences about the false-confidence tests and a list of the five risks.
Want more? Take a small function from your own project, ask your assistant to write tests for it, then break the function on purpose and see how many tests still pass.
Quiz
4 questions. Your score is saved with your progress.
1. Why should AI-generated code be treated as untrusted input?
The risk is not constant failure but plausible output that is unverified, so it needs the same scrutiny as any external input.
2. A test asserts only that a function returned a number. What is the main weakness?
A type-only check says nothing about whether the value is correct, so broken logic can still pass.
3. Which pairing of risk and example is correct?
Invented package names that attackers can register are fake packages. The other examples belong to different risks.
4. Coverage is 100% and all tests are green. What does that tell you for certain?
Coverage shows lines were executed, not that the results were checked or that the code is safe.
Finish the day
Mark the day complete to save your progress.
Next: Tomorrow we turn today's mindset into a tool: a repeatable security checklist for reading an AI-written pull request. The false-confidence habit you practised today becomes the first item on it.
Your notes
Private to this device. Write what you want to remember from today.