AI Role-Confusion Lab: Make User Text Look Like a System Rule

This article describes a role-confusion injection attack where user input can forge system-level instructions in AI systems that build prompts through plain-text concatenation of role markers. The SupportBot lab demonstrates how attackers can impersonate the SYSTEM role to extract protected information by inserting fake SYSTEM directives into their messages. The vulnerability exists because textual role boundaries offer no real separation, allowing the model to treat forged instructions with the same authority as genuine system directives.

• How the attack exploits text-based role markersWhen a system concatenates SYSTEM, USER, and ASSISTANT turns as plain text with literal role markers (like 'SYSTEM:' or 'USER:'), an attacker can include those same markers within their user message. Once concatenated into the final prompt, the forged marker appears indistinguishable from authentic role delimiters, causing the model to execute instructions that look like legitimate system directives with elevated privileges.
• Why plain-text boundaries fail fundamentallyThe core vulnerability is that role boundaries created only through text markers are not actual boundaries—they are content the attacker can also write. The model cannot differentiate a forged SYSTEM line from a genuine one once both exist in the same concatenated string, so it grants the attacker's text the same authority as real system instructions.
• Practical exploitation steps in the labThe attack involves three steps: first, confirming the prompt uses plain-text role markers that can be reproduced; second, placing a forged system-role instruction within the user message (for example, to disclose an admin-only staff PIN); third, submitting the message and observing whether the model executes the forged instruction as if it came from the system role. The concatenation process makes the fake directive indistinguishable from real ones.
• Structured chat APIs eliminate the vulnerabilityThe primary defense is using the provider's structured chat API, which enforces role boundaries outside the text itself rather than relying on text markers. This architectural approach ensures the platform, not the concatenated string, determines which instructions carry system authority, making role impersonation impossible.
• Defense-in-depth complementary measuresBeyond structured APIs, the article recommends stripping or escaping role-delimiter patterns from user input as an additional layer, and keeping secrets out of the prompt entirely so that even if a forged instruction succeeds, there is nothing sensitive to reveal.
• Classification within OWASP LLM taxonomyThe article categorizes role-confusion injection as a form of OWASP LLM01 Prompt Injection with overlap to LLM07 System Prompt Leakage, indicating it is a recognized attack class in the OWASP Top 10 for LLM Applications (2025). It is most common in hand-rolled prompt templates that predate or ignore structured chat APIs.

Bottom line: The fundamental lesson is that role boundaries enforced only through text markers are not real boundaries—they must be enforced by the platform's structured chat API architecture. Never build prompts by concatenating user input with role delimiters, as this allows attackers to forge system-level instructions that the model will execute with elevated privileges.