Day 2 of 1060 minShell WalkerHands-on lab

Day 2: Reading AI Pull Requests Like an Attacker

Taught by Mr Elite Updated

By the end of today you can

  • Read a pull request diff first and the summary second
  • Apply a four-point checklist: input, auth, errors, dependencies
  • Spot string-built queries and missing authorisation checks in generated code
  • Write each finding with a line reference and a one-line fix

Warm-up

Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.

Why does a passing test suite give false confidence about AI-written code?

Tests may not check the risky behaviour, or may be written to match the code, so they pass even when the code is wrong.

Name the five risks of AI coding tools from Day 1.

Insecure code flaws, leaked secrets, excess permissions, poisoned input and fake packages.

A pull request lands in your queue. The title is clear, the description is tidy, the tests are green and an AI agent wrote all of it. You have ten other things to do, so you skim the summary, nod and approve.

That is exactly how flawed code ships. Today I will show you how to read the diff like an attacker would, with a short checklist you can run in a few minutes, and then you will use it on a real pull request in the lab.

Read the diff, not the summary

A pull request (PR) is a proposal to merge changes into a codebase. It has two parts: the description, which says what the author claims to have done, and the diff, which shows what actually changed. An AI agent writes very good descriptions. They are fluent, confident and often slightly wrong about what the code does.

So the rule is simple. Treat the description as a claim and the diff as the evidence. Read the diff first, form your own view of what changed, then compare it with the summary. Anything the diff does that the summary never mentions deserves a closer look.

If you have not read a diff closely before, here is what one looks like. Lines starting with a plus sign were added. Lines starting with a minus sign were removed. Lines with neither are unchanged context. The block below is a made-up example.

diff
--- a/routes.js
+++ b/routes.js
@@ -10,3 +10,6 @@
+app.get('/export', (req, res) => {
+  res.json(db.all());
+});
A diff that adds a new route. Every plus line is new code that you are being asked to approve.

Notice what this small change does. It adds a route that returns every record, and nothing in it asks who is calling. A summary might call it a harmless export helper. The diff shows what it really is.

Building on yesterday, remember that green tests prove less than they appear to. In a PR, look at the test changes too. A test that only checks the happy path, or that calls a function and asserts nothing about the result, will pass whatever the code does.

Quick check

You open a PR. What should you read first?

The four-point checklist

A checklist stops you relying on instinct, which tires quickly. I use four questions, in this order. Run them over every changed file, one file at a time, rather than skimming the whole PR four times.

Input
Anything that comes from outside the code: form fields, URL parameters, headers, uploaded files. Ask where it goes and whether it is checked or cleaned on the way.
Auth
Authentication is who you are; authorisation is what you may do. Ask whether each new route or action checks both.
Errors
What happens when something fails. Ask whether the code leaks stack traces, fails open, or swallows errors silently.
Dependencies
Packages the change adds or updates. Ask whether each one is real, needed and trusted.

The order matters. Input comes first because it is where an attacker starts: if they cannot send your code anything, little else can go wrong. Auth comes next because it decides who gets to send it. Errors and dependencies come last because they are quieter, and people forget them.

You will not find a flaw in every category on every PR. The point is that you looked in each place on purpose. A review that ends with 'input fine, auth fine, errors fine, no new packages' is a real result, and worth writing down.

Mistakes AI makes again and again

Assistants copy patterns from huge amounts of public code, including bad code. Two mistakes turn up constantly. The first is the string-built query, where user input is glued straight into a SQL statement. The second is the missing check, where a route works perfectly but never asks whether the caller is allowed to use it.

javascript
const q = "SELECT * FROM users WHERE name = '" + req.query.name + "'";
db.query(q);
Dangerous: input is concatenated into SQL. A value like ' OR '1'='1 changes the meaning of the query.

To see why, picture what the database actually receives when someone sends that value as the name. The quotes balance out and the condition becomes true for every row, so the query returns the whole table.

sql
SELECT * FROM users WHERE name = '' OR '1'='1'
The final SQL after the attacker's text is glued in. The OR clause is always true, so every user comes back.
javascript
db.query("SELECT * FROM users WHERE name = ?", [req.query.name]);
Safer: a parameterised query keeps data separate from the SQL command.

With the question mark placeholder, the database receives the command and the value separately. The value is always treated as plain data, never as SQL, so there is nothing for the quotes to break out of.

For missing checks, look at every route handler and ask what stops a logged-out user, or a different user, from calling it. If the answer is nothing, that is a finding.

javascript
// No check: anyone can call this
app.delete('/user/:id', handler);

// Checked: login and role required first
app.delete('/user/:id', requireLogin, requireAdmin, handler);
The difference is the middleware in front of the handler. Look for that extra step on every route that changes data.

Quick check

Which line is the safest way to run a query using a user-supplied name?

Watch me do it

Here is a small review on a throwaway file on your own machine. No real system is involved, and you do not need to run the code, only read it. I will make a sample file, then run the checklist on it. You need a terminal: Terminal on macOS or Linux, or Git Bash or WSL on Windows.

bash
mkdir pr-practice && cd pr-practice
Makes and enters a scratch folder.
bash
cat > routes.js <<'EOF'
app.get('/user', (req, res) => {
  const q = "SELECT * FROM users WHERE id = " + req.query.id;
  db.query(q, (err, rows) => res.json(rows));
});
app.delete('/user/:id', (req, res) => {
  db.query("DELETE FROM users WHERE id = ?", [req.params.id]);
  res.send('ok');
});
EOF
Creates routes.js with deliberate flaws for us to find.
bash
grep -nE 'query(|+ req.|app.(get|post|delete)' routes.js
Prints matching lines with line numbers. -n adds the numbers, -E allows the pattern with alternatives.
text
1:app.get('/user', (req, res) => {
2:  const q = "SELECT * FROM users WHERE id = " + req.query.id;
3:  db.query(q, (err, rows) => res.json(rows));
5:app.delete('/user/:id', (req, res) => {
6:  db.query("DELETE FROM users WHERE id = ?", [req.params.id]);
Expected grep output. Lines 4 and 7 are missing because they match none of the three patterns.

Here is what I find. Input: line 2 concatenates req.query.id into SQL, so that is injection. Line 6 is fine, because it uses a placeholder. Auth: the delete route on line 5 has no login or role check, so anyone can delete any user. The read route on line 1 has none either.

Errors: line 3 ignores err, and if the query fails the code returns undefined rows with no handling. Line 7 also replies ok even if the delete failed. Dependencies: nothing new here, so I note that and move on. Four categories, four findings, about two minutes.

Notice that grep only pointed me at places to look. It could not tell me line 5 was missing a check, because a missing check leaves nothing to match. That finding came from reading, which is why the checklist matters more than the search.

javascript
app.get('/user', requireLogin, (req, res) => {
  db.query("SELECT * FROM users WHERE id = ?", [req.query.id], (err, rows) => {
    if (err) return res.status(500).send('error');
    res.json(rows);
  });
});
A corrected version of the first route: login required, parameterised query, and the error handled without leaking details.

Your turn: the lab

Now you will do this on a real PR. The lab below gives you an AI-written pull request that passes its tests and reads well. Five lines in it are planted flaws. Use the checklist, read the diff before the summary, and keep notes as you go.

Do not worry if you find three quickly and then stall. That is normal. The last flaws tend to sit in the places people stop looking, such as test files or small side changes. Give yourself about twenty minutes.

Your turn

Review the AI's Pull Request

This lab is a safe, contained pull request built to look like ordinary AI output. It lets you practise reading a diff under realistic conditions, where everything looks fine until you check each line against the checklist.

Open the lab in a new tab

What to try

What to notice

  • How confident and tidy the description sounds compared with what the code does
  • Which flaws hide in files that look unrelated to the main change
  • Whether the tests touch the risky lines at all

If it did not work

Approving because tests are green

Open the test diff and ask whether any test would fail if the risky line were broken. If not, the green tick tells you nothing.

Reading only the files the description mentions

Scan the full file list in the diff. Flaws often sit in a small side change the summary skips.

Writing vague findings such as 'looks insecure'

Record file, line, the specific problem and a one-line fix so someone else can act on it.

Stopping after the first flaw

Finish all four checklist categories. Planted or accidental flaws rarely come alone.

Relying on grep to find everything

Use grep to point at places to read, not as proof of safety. Missing checks leave nothing to match, so only reading finds them.

Recap

  • Treat the PR description as a claim and the diff as evidence
  • Run input, auth, errors, dependencies on every changed file
  • String-built queries and missing authorisation checks are the most common AI flaws
  • Review the tests too, because weak tests hide broken code
  • Good findings give file, line, problem and a one-line fix

Cheat sheet

Input
Where outside data enters and whether it is validated or parameterised
Auth
Does each route check who the caller is and what they may do
Errors
No leaked traces, no silent failures, no fail-open behaviour
Dependencies
Every new package is real, needed and trusted
grep -nE 'pattern' file
Search a file with line numbers to find risky patterns fast
Parameterised query
db.query(sql, [value]) keeps data apart from SQL code
+ and - in a diff
Plus lines were added, minus lines were removed

Before the next day

Mission: Planted flaws report

Finish the lab and write up what you found. A written finding is the habit that makes reviews repeatable, and you will reuse this format in later days when you audit code, secrets and packages.

About 30 min

You are done when: A file pr-findings.md listing five flaws, each with line reference, category and a one-line fix.

Want more? Run the same checklist on a recent PR in one of your own repositories and note one thing you would now ask the author to change.

Quiz

4 questions. Your score is saved with your progress.

1. An AI-written PR has a clear summary and passing tests. What is the best next step?

2. A new route deletes records and the handler contains no user check. Which category is this?

3. Why is concatenating req.query.id into a SQL string dangerous?

4. Which finding is written well enough to act on?

Finish the day

Mark the day complete to save your progress.

Next: Day 3 is module practice. You will take a classic injection flaw in code an assistant wrote and fix it yourself, using the string-built query pattern you spotted today. Unlocks 17 Nov.

Tell someone you are doing this:WhatsAppXLinkedIn

Your notes

Private to this device. Write what you want to remember from today.