AI Token Forgery Lab: When the Backend Trusts a Model-Made Token

This article describes a critical authentication design flaw where a backend system trusts session tokens generated directly by a language model within its responses. Because LLMs are text generators that can be influenced through prompting, an attacker can steer the model to emit a forged token with elevated privileges like admin role, which the backend then accepts as legitimate. The vulnerability exemplifies how improperly integrating LLMs into security-critical workflows creates a false trust boundary.

• The core design errorAuthBot's backend accepts session tokens that the LLM writes into its response text after login, treating the model as an identity provider. The token's claims—including the role field—are whatever the model outputs, which an attacker can influence through prompt steering, making the token's security guarantees meaningless.
• Why LLMs cannot provide authenticationModels generate text and can be influenced by input; they cannot hold a signing key secret or guarantee a claim remains unforged. Unlike a real auth service, an LLM has no cryptographic mechanism to bind a token to an actual authentication event, so any token it produces is as trustworthy as any sentence it can be talked into writing.
• Attack execution in practiceAn attacker confirms the token in AuthBot's reply comes from the model and is trusted by the backend, then steers the model to emit a token claiming admin role instead of user role. The attacker copies that forged token and submits it to an admin-only endpoint, which the backend honors because it trusts the LLM's output, granting access to protected resources like the user list.
• No cryptographic binding existsThe backend accepts a token claiming admin status simply because it came back from the LLM's response, with nothing cryptographically linking the token to a real authentication event. This differs from legitimate JWT-based systems where a signing key creates a verifiable chain of trust that the model cannot compromise.
• Real-world security classificationThis vulnerability is categorized as OWASP LLM05 Improper Output Handling and overlaps with Excessive Agency. As teams integrate LLMs into auth and authorization flows, treating 'the model said so' as a security decision becomes a direct attack vector rather than a trust boundary.
• Defense requires auth service ownershipTokens must be issued and signed by the auth service using a key the model never sees or accesses. Role and scope claims must be set by the backend, not derived from the model's reply, and all authorization decisions must remain outside the model entirely. Server-side verification must check tokens against the issuer and validate claims independently.
• Conceptual overlap with JWT attacksLLM token forgery mirrors JWT vulnerabilities like algorithm bypass and algorithm confusion, which also stem from trusting a token's claims without real cryptographic guarantees tied to an authentication event. Both expose the danger of accepting claims at face value rather than verifying them through proper signature validation.

Bottom line: Never allow an LLM to produce anything a backend trusts for security decisions—tokens, roles, or approval flags—because models generate text you can influence, not cryptographic proofs you cannot forge. Authentication state must originate and be verified exclusively by services the model cannot reach or manipulate.