Prompt Injection Lab: Make a Sales Bot Leak Its Secret

This article presents a beginner-level prompt injection lab that demonstrates how language models can be manipulated to reveal secrets despite explicit instructions not to. The lab uses a retail chatbot programmed to never disclose a staff discount code, yet the code can be easily extracted through social engineering rather than technical exploits. The article explains why prompt injection works, shows real-world examples, and provides practical defenses, arguing that system prompts alone are insufficient security controls.

• System prompts lack enforceability boundariesThe article explains that language models read system prompts and user messages as a single stream of text with no hardware or logical separation between developer rules and user input. When these conflict, the model makes a judgment call that users can steer, meaning the bot must choose between competing goals—following the rule versus being helpful—without verification mechanisms to confirm user identity or intent.
• Social engineering trumps technical defensesThe RetailBot attack succeeds not through special characters, code tricks, or phrases like 'ignore previous instructions,' but through a believable narrative: the attacker claims to be the store owner conducting a configuration audit and asks the bot to read back its setup. This works because the model has no way to verify the requester's actual identity or authorization level.
• Helpfulness and compliance create competing goalsThe article identifies a fundamental conflict in LLM design: the system prompt tells the bot to never reveal the code, but it also tells it to help customers. User-supplied context that adds plausibility makes the model resolve this conflict in favor of helpfulness. The bot prioritizes being useful to whoever claims authority rather than enforcing its security rule.
• Real-world precedents show practical riskThe article cites two documented incidents: in 2023, users extracted Bing Chat's hidden instructions and internal codename simply by asking it to repeat text above the conversation, and a car dealership's ChatGPT-powered assistant was socially engineered into 'agreeing' to sell an SUV for one dollar. Both exploits relied on conversation, not code manipulation.
• Secrets should never reside in model contextThe article's primary defense recommendation is architectural: do not store sensitive values in prompts at all. Instead, fetch secrets server-side after performing real authorization checks outside the model, and filter outputs to block known secret values from leaving the system. Layering separate classifiers to judge user intent adds further protection against social engineering.
• Prompt injection is user-guided redirectionThe article clarifies that prompt injection is not a code bug but rather the art of writing text that makes the model decide breaking its rule is helpful and reasonable. This reframes the attack as text engineering rather than technical exploitation, which is why social engineering tactics prove more effective than technical tricks.
• Lab emphasizes testing before productionThe article concludes with an elite-level recommendation: before launching an LLM feature, run this exact attack against it yourself. If a one-line social engineering message extracts sensitive information, assume a determined attacker will extract more. This positions prompt injection testing as a mandatory pre-deployment security activity.

Bottom line: Prompt injection succeeds because system prompts are requests, not access controls—the model processes developer instructions and user input identically, allowing attackers to resolve conflicts between rules and helpfulness through credible social engineering rather than technical tricks. Effective defense requires keeping secrets out of the model entirely, verifying identity server-side, and filtering outputs, not relying on the model to enforce restrictions alone.