Day 3 of 760 minCipher CadetHands-on lab
Day 3: Keep Secrets Out and Check the AI's Answers
Taught by Mr Elite Updated
By the end of today you can
- List the kinds of secret that must never be pasted into a public AI chat
- Redact keys, passwords, tokens and personal details from a log or config with placeholders
- Spot hallucination and false confidence in an AI answer
- Verify an AI claim against a trusted source and record what you found
Warm-up
Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.
In the role, context, task and format pattern, what does the context part give the model?
The information it cannot see for itself, such as the log line, the system involved or what you already know.
Where should instructions go relative to the pasted data, and why?
Instructions first, pasted data last, so the model knows what to do before it reads the material and the data is not mistaken for orders.
Picture someone tired and under pressure. An app keeps crashing, they copy the whole error output and the config file, paste it into a chatbot and ask for a fix. The answer is good. What they did not notice is that the paste held a database password, an API key and a customer email address. All of it has now left their machine.
Today has two halves. First you learn to clean text before it goes anywhere near an AI. Then you learn the other habit that goes with it: never trust the answer that comes back until you have checked it. Both skills are quick, and both stop real problems.
What counts as a secret
A secret is any piece of text that gives someone access, or identifies someone, and would cause harm if a stranger had it. Many beginners think only passwords count. In practice the list is longer, and the dangerous items are often buried in the middle of a long paste.
- API key
- A long string that proves your program is allowed to use a service. Whoever holds it can use the service as you, and you may pay the bill.
- Connection string
- A line that tells software how to reach a database, often with the username and password inside it.
- PII
- Personally identifiable information: names, emails, phone numbers, addresses, anything that points to a real person.
Add to that list private IP addresses and internal hostnames, private keys (the blocks starting with BEGIN PRIVATE KEY), session cookies, and anything your employer marks as confidential. A public chatbot may store what you type, and staff or systems may review it. Treat every paste as if it will be read by someone you never met.
Quick check
Which of these is the most dangerous to paste into a public AI chat?
The connection string contains a working credential that gives direct access to your data.
Redacting: replace, do not delete
Redacting means swapping the sensitive part for a placeholder. I say replace rather than delete because the model still needs the shape of the text to help you. If you cut a line out, the answer may no longer make sense. If you write a clear label, the structure stays and the secret goes.
Use labels that say what was removed, and keep them consistent. If the same key appears twice, use the same placeholder twice, so the model can see they match.
Before:
DB_URL=postgres://admin:Sup3rS3cret@10.0.4.12:5432/shop
API_KEY=sk_live_9f8a7b6c5d4e3f2a1b0c
user jane.smith@example.com failed login
After:
DB_URL=postgres://<DB_USER>:<DB_PASSWORD>@<DB_HOST>:5432/shop
API_KEY=<API_KEY>
user <EMAIL_1> failed loginWatch me do it: redact a paste
I will clean a small made-up log the way I would before asking for help. Everything here is fake sample text, so you can follow along in your own text editor. Nothing touches a real system.
2025-03-02 09:14:07 login failed user=jane.smith@example.com
2025-03-02 09:14:07 source=192.168.1.40 password=Winter2025!
2025-03-02 09:14:09 retry with header Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.abc123
2025-03-02 09:14:10 upstream db at 10.0.4.12 refused connection
2025-03-02 09:14:11 login failed user=jane.smith@example.com2025-03-02 09:14:07 login failed user=<EMAIL_1>
2025-03-02 09:14:07 source=<INTERNAL_IP_1> password=<PASSWORD>
2025-03-02 09:14:09 retry with header Authorization: Bearer <TOKEN>
2025-03-02 09:14:10 upstream db at <INTERNAL_IP_2> refused connection
2025-03-02 09:14:11 login failed user=<EMAIL_1>What just happened: the AI can still tell you that repeated failures came from one user, that a token was sent and that the database refused a connection. None of the dangerous values left your machine.
Quick check
Why do we use the same placeholder, such as EMAIL_1, for repeated values?
Consistent labels preserve relationships in the data, which the model needs to reason well.
Hallucination and false confidence
Now the other half. An AI model predicts likely text. It does not look facts up the way a search engine does. Sometimes it produces something that sounds right and is not. That is called a hallucination: an invented fact, command, function name or reference, delivered in a calm voice.
False confidence is related. The model rarely says I am not sure. A wrong answer and a right answer often read exactly the same. In security work this matters, because a made-up command flag or a wrong claim that something is safe can waste hours or leave a hole open.
Two kinds of mistake to watch for today. First, invented details, such as a tool option, CVE number or setting that does not exist. Second, work that looks complete but proves nothing, such as tests that pass whatever the code does. You will meet the second kind in the lab.
Verify against a trusted source
Verifying means checking a claim somewhere the AI does not control. Good sources are official documentation, the vendor's own site, the manual page on your machine, or the actual text you pasted in. Another AI answer is not a trusted source, because it can be wrong in the same way.
ping --helpQuick check
An AI tells you a command has a flag called --safe-mode. What is the best next step?
Only a source outside the model can confirm the flag exists; repetition from the same model proves nothing.
Today's labs
Two labs, one for each half of the day. Both are SecurityElites practice labs, so you work on provided sample material and touch no real system.
Your turn
Secrets in the Context Window
This lab puts the redaction habit under pressure: a stack trace, a config snippet and logs, all about to go into a public chatbot. Start here, then do the second lab, ai-dev-fake-tests-1, where you catch an AI that looks confident but proves nothing. Log in to your SecurityElites account to open both.
What to try
What to notice
- How many secrets hide inside text that looks like harmless debugging output
- That a redacted paste still makes sense and is still useful
- That 100% coverage and green ticks do not mean the tests check anything
If it did not work
Redacting only the obvious password and missing tokens, emails or internal hostnames
Do two passes: one search by keyword, then one slow read of the full text before sending.
Deleting whole lines so the AI loses the context it needs
Replace the value with a labelled placeholder and keep the line.
Believing an answer because it sounds confident
Pick one concrete claim and check it in official documentation or against your own text.
Using another AI to verify the first AI
Use a source the model does not control, such as docs, a manual page or the original file.
Recap
- A secret is anything that grants access or identifies a person, not only passwords
- Replace secrets with consistent labelled placeholders so the structure survives
- Assume anything pasted into a public AI could be stored, so redact first
- Hallucinations are invented facts delivered with confidence
- Asking for step-by-step reasoning helps you find weak spots but does not replace checking
- Verify against official documentation or the original text, and write the result down
Cheat sheet
<PASSWORD>, <API_KEY>, <TOKEN>- Placeholder labels that replace real secret values
EMAIL_1, EMAIL_2- Numbered placeholders that keep repeated values linked
Hallucination- An invented fact or command presented as true
False confidence- A wrong answer that sounds as sure as a right one
Rotate a secret- Create a new one and disable the leaked one
Explain step by step and mark uncertainty- A follow-up that exposes weak reasoning
tool --help- Built-in help on your own machine for checking options
Before the next day
Mission: Redact, run, catch
Finish both labs, then write a short note that proves you can protect a paste and challenge an answer. This is the habit that keeps your later AI work safe, and it prepares you for attacking and defending AI systems from tomorrow.
About 30 min
You are done when: A saved day3.txt listing the secret types you redacted and one AI mistake you caught, with the source you used to verify it.
Want more? Ask an AI assistant to explain a command option, then check it against the tool's own --help output or manual page, and record whether it was right.
Quiz
4 questions. Your score is saved with your progress.
1. Why is replacing a password with better than deleting the line?
The structure gives the model context to help, while the real value stays with you.
2. You pasted a live API key into a public chatbot by accident. What is the right response?
You cannot confirm deletion, so you treat the key as leaked and replace it.
3. Which best describes a hallucination?
Hallucinations are made-up content that reads as confident and plausible.
4. AI-written tests give 100% coverage. Why might you still distrust them?
Coverage shows code was run, not that results were checked, so a weak test can pass even when the code is broken.
Finish the day
Mark the day complete to save your progress.
Next: Day 4 turns the lens around: instead of protecting what you send to an AI, you see how text sent to it can act as orders. You will meet prompt injection and try it on the practice sales bot.
Your notes
Private to this device. Write what you want to remember from today.