Day 6 of 1260 minCipher CadetHands-on lab
Day 6: Supply Chain Risk: Models, Packages and Plugins
Taught by Mr Elite Updated
By the end of today you can
- List the parts of an AI application supply chain and say how each can be poisoned
- Explain package hallucination and slopsquatting in plain words
- Check a package's name, owner, age and download history on a registry
- Explain why pinning and reviewing matter and write a pinned requirements file
Warm-up
Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.
What is the context window?
Everything the model can see in one request: your prompt, earlier messages, pasted files and any retrieved text.
If a real API key has been pasted into a public AI tool, what should you do?
Treat it as exposed and rotate it, meaning revoke it and issue a new one.
You ask an AI coding assistant to set up a small web API. It replies in seconds with a neat install command listing seven packages. It looks tidy and confident, and one paste would have it all running.
Here is the catch. Some of those names may not be what they seem. One may not exist until an attacker registers it. Another may be a lookalike of a famous library. Today I will show you how to slow down for two minutes and check, because that pause is the whole defence.
What an AI supply chain contains
A supply chain is everything you bring in from other people to build your product. You did not write it, but you run it with your own permissions. An LLM application has more of these than most software, and each one is a way in for an attacker.
- Model weights
- The big file of numbers that is the trained model. If someone tampers with it, the model can behave badly in ways that are hard to see.
- Package
- A bundle of code, such as a Python library from PyPI or a JavaScript one from npm, that you install with one command.
- Plugin
- An add-on that gives an assistant extra abilities, such as reading email or calling a web service.
- Dependency
- Any package your project needs to run. Dependencies have their own dependencies, so the list grows quickly.
- Registry
- The public catalogue where packages are published, such as pypi.org or npmjs.com. Anyone can publish there.
The danger is simple. Installing a package can run its code on your machine straight away, with your files and your keys in reach. A poisoned model from a download site, a trojaned package or a rogue plugin all arrive the same way: you trusted a name.
Let me make each one concrete. A model file in some older formats can contain code that runs the moment you load it, so a tampered download can act before you ever type a prompt. A package can carry an install script that reads environment variables, where API keys often live. A plugin can ask for access to your mailbox or files, and a careless yes hands all of it over.
Quick check
Why is installing a package riskier than reading about one?
Install scripts and imported code run as you, so a malicious package can reach your files and keys.
Package hallucination and slopsquatting
A hallucination is when a model states something that is not true, confidently. With code, that includes inventing package names that sound right but do not exist. Models do this fairly often, and often they repeat the same invented name for many different users.
Slopsquatting is what attackers do with that. They notice a name an AI keeps inventing, register it on the registry, and fill it with harmful code. The next person who pastes the AI's install command gets the real, malicious package. Nothing was hacked. The assistant simply sent you to a trap.
Here is the sequence, so you can spot it. First the model suggests a name that sounds plausible, such as flask-jwt-helper-pro. Second, nobody owns that name yet. Third, an attacker registers it and uploads code that looks harmless on the surface. Fourth, a developer pastes the command without checking, and the install runs the attacker's code.
Other warning signs are lookalike names (a famous library with an extra word or swapped letters), brand new packages with almost no users, and packages whose description does not match what you asked for.
There is a trap in the other direction too. A name that exists is not a name that is safe. Once an attacker has registered a hallucinated name, a search will find it. That is why we check four separate signals rather than one.
Watch me do it: vetting one package
I will check one name on the Python registry. Everything here is read-only. We are looking, not installing. Use the name requests as our known good example, then repeat the same steps on any name an AI gives you. First the browser method, then a quick command-line version.
Now the same checks from a terminal on your own machine. PyPI publishes a public JSON page for every package. This command only reads that page and prints three facts: the name, the newest version and the earliest upload date.
curl -s https://pypi.org/pypi/requests/json | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['info']['name'], d['info']['version']); print(min(f['upload_time'] for v in d['releases'].values() for f in v))"You should see the name requests, a version number such as 2.x.x, and an earliest date many years in the past. If the command prints nothing or an error, check that you have internet access and that python3 is installed by running python3 --version.
Next, see what a missing name looks like. This asks only for the HTTP status code, the number the server replies with. A real package returns 200. A name nobody has registered returns 404.
curl -s -o /dev/null -w "%{http_code}n" https://pypi.org/pypi/requests/json
curl -s -o /dev/null -w "%{http_code}n" https://pypi.org/pypi/zz-made-up-name-48213/jsonExpect 200 on the first line and 404 on the second. A 404 on an AI-suggested name tells you the model invented it. Do not register it, and do not install it. Just reject the suggestion and look for a real library.
Pinning and reviewing what you install
Pinning means fixing the exact version you checked, so an update cannot silently swap in something new. If you write requests without a version, tomorrow you might receive a different release than the one you reviewed. A pinned line looks like this.
requests==2.31.0
flask==3.0.0Let me show you how long a list really gets. We will use a virtual environment, which is a throwaway folder of Python packages kept separate from the rest of your machine. We install only requests, a package we have just vetted, and then look at what came with it.
Review means reading the list before you run it. Ask four things: do I recognise each name, does each one have a reason to be there, did I check the ones I did not know, and is anything doing a job I did not ask for? A web API does not need a package called crypto-wallet-helper.
The same habit applies beyond code. For models, download from the publisher's official page, check who uploaded it, and prefer formats that cannot run code on load. For plugins, read what permissions they ask for and refuse any that want more than the task needs.
Quick check
An AI suggests a package. The registry shows it was created four days ago with 12 downloads. What is the best action?
A very new package with almost no users fits the slopsquatting pattern, so find an established one and verify that.
Lab: Check Before You Install
Now you do it on a realistic case. The lab gives you an AI's install command with seven packages. Three should not be installed. Your job is to find which ones and prove it with evidence.
The lab runs in the SecurityElites practice environment, so nothing you look up touches a real registry or your own machine. Start the lab named Check Before You Install, read the install command through once, and then work down the list one name at a time using the same four checks from the worked example.
Keep a plain text file open beside the lab as you go. Write each name, then the four findings, then your decision. If you get stuck on one package, move on and return to it. Comparing it with the others often shows what a normal entry looks like.
Your turn
Check Before You Install
This lab recreates the exact moment from the start of today: an AI hands you a confident install command and you decide what to trust. It runs in the SecurityElites practice environment, so nothing touches a real registry or your machine.
What to try
What to notice
- How convincing a bad name can look until you compare it character by character
- That age and download count often tell you more than the description
- That a package can look fine on one check and fail another, so one signal is not enough
If it did not work
Trusting a package because the AI sounded certain
Confidence is not evidence. Check the registry for every name you do not already know, every time.
Checking only whether the page exists
Attackers register hallucinated names, so a page proves nothing. Check owner, age, downloads and linked source.
Leaving versions unpinned
Write exact versions with == after you have reviewed them, and review again when you update.
Only recording the rejects
Write evidence for the kept packages too. It shows you checked rather than guessed.
Recap
- An AI supply chain includes models, packages, plugins and the data they bring with them
- Installing runs someone else's code with your permissions
- Slopsquatting turns an AI's invented package name into a real malicious package
- Check name, owner, age and download history before you install anything unfamiliar
- Pin exact versions and review the whole list before you run it
- Write down evidence for every decision
Cheat sheet
Slopsquatting- Registering a package name an AI tends to invent, then filling it with malicious code
Typosquatting- Registering a name close to a popular one, hoping for typing mistakes
Four checks- Name, owner, age, download history
pypi.org- Public Python package registry; shows release history and maintainers
pepy.tech- Free site showing download counts for Python packages
package==1.2.3- Pins an exact version in requirements.txt
Registry 404- The name does not exist yet, so an attacker could claim it
pip freeze- Lists every installed package with its exact version, including dependencies you did not ask for
Before the next day
Mission: Dependency check report
Finish the lab, then write a short report of what you rejected and why. This is the habit you will reuse in the capstone audit, and it is the evidence a reviewer would want to see.
About 30 min
You are done when: day6-dependency-check.txt with seven decisions and evidence, plus a screenshot of the completed lab
Want more? Ask any AI assistant to suggest packages for a small project of your choice, then vet every name on pypi.org or npmjs.com without installing anything, and record whether any name did not exist.
Quiz
4 questions. Your score is saved with your progress.
1. What makes slopsquatting different from ordinary typosquatting?
The attacker registers a name that models keep hallucinating, so victims arrive by following the AI's advice.
2. A package page exists and its description matches what you wanted. Why is that not enough to trust it?
Anything on a registry can be copied or faked, so independent signals are the real evidence.
3. What does pinning protect you from?
Pinning fixes the exact version you reviewed, so updates cannot silently change what you run.
4. An AI-suggested list for a simple web API includes a package that handles cryptocurrency wallets. What should you conclude?
A dependency with no reason to be there is suspicious, whatever it looks like.
Finish the day
Mark the day complete to save your progress.
Next: Day 7 moves from what you install to what the model learns from and retrieves. You will see how bad data in a training set or knowledge base changes answers, and demonstrate it in a lab.
Your notes
Private to this device. Write what you want to remember from today.