Day 2 of 1060 minShell WalkerHands-on lab
Day 2: Reading AI Pull Requests Like an Attacker
Taught by Mr Elite Updated
By the end of today you can
- Read a pull request diff first and the summary second
- Apply a four-point checklist: input, auth, errors, dependencies
- Spot string-built queries and missing authorisation checks in generated code
- Write each finding with a line reference and a one-line fix
Warm-up
Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.
Why does a passing test suite give false confidence about AI-written code?
Tests may not check the risky behaviour, or may be written to match the code, so they pass even when the code is wrong.
Name the five risks of AI coding tools from Day 1.
Insecure code flaws, leaked secrets, excess permissions, poisoned input and fake packages.
A pull request lands in your queue. The title is clear, the description is tidy, the tests are green and an AI agent wrote all of it. You have ten other things to do, so you skim the summary, nod and approve.
That is exactly how flawed code ships. Today I will show you how to read the diff like an attacker would, with a short checklist you can run in a few minutes, and then you will use it on a real pull request in the lab.
Read the diff, not the summary
A pull request (PR) is a proposal to merge changes into a codebase. It has two parts: the description, which says what the author claims to have done, and the diff, which shows what actually changed. An AI agent writes very good descriptions. They are fluent, confident and often slightly wrong about what the code does.
So the rule is simple. Treat the description as a claim and the diff as the evidence. Read the diff first, form your own view of what changed, then compare it with the summary. Anything the diff does that the summary never mentions deserves a closer look.
If you have not read a diff closely before, here is what one looks like. Lines starting with a plus sign were added. Lines starting with a minus sign were removed. Lines with neither are unchanged context. The block below is a made-up example.
--- a/routes.js
+++ b/routes.js
@@ -10,3 +10,6 @@
+app.get('/export', (req, res) => {
+ res.json(db.all());
+});Notice what this small change does. It adds a route that returns every record, and nothing in it asks who is calling. A summary might call it a harmless export helper. The diff shows what it really is.
Building on yesterday, remember that green tests prove less than they appear to. In a PR, look at the test changes too. A test that only checks the happy path, or that calls a function and asserts nothing about the result, will pass whatever the code does.
Quick check
You open a PR. What should you read first?
The diff is the evidence of what changed; the description is only a claim to compare against it.
The four-point checklist
A checklist stops you relying on instinct, which tires quickly. I use four questions, in this order. Run them over every changed file, one file at a time, rather than skimming the whole PR four times.
- Input
- Anything that comes from outside the code: form fields, URL parameters, headers, uploaded files. Ask where it goes and whether it is checked or cleaned on the way.
- Auth
- Authentication is who you are; authorisation is what you may do. Ask whether each new route or action checks both.
- Errors
- What happens when something fails. Ask whether the code leaks stack traces, fails open, or swallows errors silently.
- Dependencies
- Packages the change adds or updates. Ask whether each one is real, needed and trusted.
The order matters. Input comes first because it is where an attacker starts: if they cannot send your code anything, little else can go wrong. Auth comes next because it decides who gets to send it. Errors and dependencies come last because they are quieter, and people forget them.
You will not find a flaw in every category on every PR. The point is that you looked in each place on purpose. A review that ends with 'input fine, auth fine, errors fine, no new packages' is a real result, and worth writing down.
Mistakes AI makes again and again
Assistants copy patterns from huge amounts of public code, including bad code. Two mistakes turn up constantly. The first is the string-built query, where user input is glued straight into a SQL statement. The second is the missing check, where a route works perfectly but never asks whether the caller is allowed to use it.
const q = "SELECT * FROM users WHERE name = '" + req.query.name + "'";
db.query(q);To see why, picture what the database actually receives when someone sends that value as the name. The quotes balance out and the condition becomes true for every row, so the query returns the whole table.
SELECT * FROM users WHERE name = '' OR '1'='1'db.query("SELECT * FROM users WHERE name = ?", [req.query.name]);With the question mark placeholder, the database receives the command and the value separately. The value is always treated as plain data, never as SQL, so there is nothing for the quotes to break out of.
For missing checks, look at every route handler and ask what stops a logged-out user, or a different user, from calling it. If the answer is nothing, that is a finding.
// No check: anyone can call this
app.delete('/user/:id', handler);
// Checked: login and role required first
app.delete('/user/:id', requireLogin, requireAdmin, handler);Quick check
Which line is the safest way to run a query using a user-supplied name?
Parameters keep user data out of the SQL syntax, so it cannot change the query.
Watch me do it
Here is a small review on a throwaway file on your own machine. No real system is involved, and you do not need to run the code, only read it. I will make a sample file, then run the checklist on it. You need a terminal: Terminal on macOS or Linux, or Git Bash or WSL on Windows.
mkdir pr-practice && cd pr-practicecat > routes.js <<'EOF'
app.get('/user', (req, res) => {
const q = "SELECT * FROM users WHERE id = " + req.query.id;
db.query(q, (err, rows) => res.json(rows));
});
app.delete('/user/:id', (req, res) => {
db.query("DELETE FROM users WHERE id = ?", [req.params.id]);
res.send('ok');
});
EOFgrep -nE 'query(|+ req.|app.(get|post|delete)' routes.js1:app.get('/user', (req, res) => {
2: const q = "SELECT * FROM users WHERE id = " + req.query.id;
3: db.query(q, (err, rows) => res.json(rows));
5:app.delete('/user/:id', (req, res) => {
6: db.query("DELETE FROM users WHERE id = ?", [req.params.id]);Here is what I find. Input: line 2 concatenates req.query.id into SQL, so that is injection. Line 6 is fine, because it uses a placeholder. Auth: the delete route on line 5 has no login or role check, so anyone can delete any user. The read route on line 1 has none either.
Errors: line 3 ignores err, and if the query fails the code returns undefined rows with no handling. Line 7 also replies ok even if the delete failed. Dependencies: nothing new here, so I note that and move on. Four categories, four findings, about two minutes.
Notice that grep only pointed me at places to look. It could not tell me line 5 was missing a check, because a missing check leaves nothing to match. That finding came from reading, which is why the checklist matters more than the search.
app.get('/user', requireLogin, (req, res) => {
db.query("SELECT * FROM users WHERE id = ?", [req.query.id], (err, rows) => {
if (err) return res.status(500).send('error');
res.json(rows);
});
});Your turn: the lab
Now you will do this on a real PR. The lab below gives you an AI-written pull request that passes its tests and reads well. Five lines in it are planted flaws. Use the checklist, read the diff before the summary, and keep notes as you go.
Do not worry if you find three quickly and then stall. That is normal. The last flaws tend to sit in the places people stop looking, such as test files or small side changes. Give yourself about twenty minutes.
Your turn
Review the AI's Pull Request
This lab is a safe, contained pull request built to look like ordinary AI output. It lets you practise reading a diff under realistic conditions, where everything looks fine until you check each line against the checklist.
What to try
What to notice
- How confident and tidy the description sounds compared with what the code does
- Which flaws hide in files that look unrelated to the main change
- Whether the tests touch the risky lines at all
If it did not work
Approving because tests are green
Open the test diff and ask whether any test would fail if the risky line were broken. If not, the green tick tells you nothing.
Reading only the files the description mentions
Scan the full file list in the diff. Flaws often sit in a small side change the summary skips.
Writing vague findings such as 'looks insecure'
Record file, line, the specific problem and a one-line fix so someone else can act on it.
Stopping after the first flaw
Finish all four checklist categories. Planted or accidental flaws rarely come alone.
Relying on grep to find everything
Use grep to point at places to read, not as proof of safety. Missing checks leave nothing to match, so only reading finds them.
Recap
- Treat the PR description as a claim and the diff as evidence
- Run input, auth, errors, dependencies on every changed file
- String-built queries and missing authorisation checks are the most common AI flaws
- Review the tests too, because weak tests hide broken code
- Good findings give file, line, problem and a one-line fix
Cheat sheet
Input- Where outside data enters and whether it is validated or parameterised
Auth- Does each route check who the caller is and what they may do
Errors- No leaked traces, no silent failures, no fail-open behaviour
Dependencies- Every new package is real, needed and trusted
grep -nE 'pattern' file- Search a file with line numbers to find risky patterns fast
Parameterised query- db.query(sql, [value]) keeps data apart from SQL code
+ and - in a diff- Plus lines were added, minus lines were removed
Before the next day
Mission: Planted flaws report
Finish the lab and write up what you found. A written finding is the habit that makes reviews repeatable, and you will reuse this format in later days when you audit code, secrets and packages.
About 30 min
You are done when: A file pr-findings.md listing five flaws, each with line reference, category and a one-line fix.
Want more? Run the same checklist on a recent PR in one of your own repositories and note one thing you would now ask the author to change.
Quiz
4 questions. Your score is saved with your progress.
1. An AI-written PR has a clear summary and passing tests. What is the best next step?
Summaries and tests can both mislead. The diff is the evidence, and the checklist makes you examine it systematically.
2. A new route deletes records and the handler contains no user check. Which category is this?
Nothing verifies who is calling or whether they are allowed, which is an authorisation gap.
3. Why is concatenating req.query.id into a SQL string dangerous?
Attacker-controlled text becomes part of the SQL syntax, so it can alter what the query does.
4. Which finding is written well enough to act on?
It names the file and line, the problem and the fix, so anyone can verify and repair it.
Finish the day
Mark the day complete to save your progress.
Next: Day 3 is module practice. You will take a classic injection flaw in code an assistant wrote and fix it yourself, using the string-built query pattern you spotted today. Unlocks 17 Nov.
Your notes
Private to this device. Write what you want to remember from today.