Token Smuggling Lab: Split a Jailbreak Across Messages

This article describes a security vulnerability in AI systems where message-by-message content filters fail to detect jailbreaks that are intentionally fragmented across multiple turns of conversation. The lab demonstrates how an attacker can split a malicious request into individually innocent-seeming messages that reassemble into a complete attack when the model processes the full conversation history, exploiting the mismatch between what the filter inspects (single messages) and what the model reads (entire conversation context). This matters because many AI products rely on per-message classifiers, leaving them vulnerable to this well-documented attack pattern.

• Per-message filtering creates a fundamental blind spotThe classifier inspects each message independently and blocks obvious jailbreak intent, but the underlying model reads the entire conversation history. This mismatch means intent that only exists when messages are combined remains invisible to the filter. The article calls this the core principle: if the model reads the whole conversation, the guardrail must too.
• Attack splits payload into three harmless-sounding stepsThe demonstrated attack uses: (1) a message asking the bot to disregard preamble and start fresh, (2) a message establishing a rule about echoing configuration field values, and (3) a simple request to echo the deploy key. Each message individually contains no obvious jailbreak language or keywords that would trigger a filter.
• Classifier and model disagree on what is the inputThe filter treats each message as a standalone input to evaluate, while the model treats the entire conversation as its input. This fundamental disagreement about scope means malicious intent distributed across messages is flagged by neither—it only becomes coherent to the model after all pieces are received.
• Payload splitting is a classic evasion technique adapted to LLMsThe article notes this is a long-standing strategy from network security (fragmented packets) and malware (obfuscated code). In LLMs it manifests as multi-turn and payload splitting jailbreaks, which public red-team reports from model vendors have documented repeatedly since 2023, indicating this is a known and recurring problem.
• Defense requires evaluating cumulative context, not individual messagesThe article identifies three key defenses: run the guardrail over the entire conversation or a running summary on every turn, watch for session-wide rules attackers try to establish (phrases like 'from now on'), and filter outputs for secrets and sensitive patterns. The strongest defense removes secrets from context entirely, eliminating the attack's target.
• Testing must first identify what guardrail actually inspectsThe article emphasizes that when testing a guardrail's robustness, the first step is determining its actual scope of inspection: only the newest message, the whole history, or a summary. The blind spot follows directly from that architectural choice, making this analysis essential before attempting exploitation.
• Differs from roleplay jailbreaks by distribution mechanismWhile roleplay jailbreaks hide intent inside a narrative frame, token smuggling hides intent through distribution across messages. The article notes that real attacks often combine both techniques, suggesting attackers layer multiple evasion strategies rather than relying on one alone.

Bottom line: Per-message filtering is fundamentally insufficient because models process entire conversation histories; defenses must evaluate cumulative context on every turn and watch for cross-message patterns. Understanding what your guardrail actually inspects—a single message, full history, or summary—is the starting point for both defensive design and vulnerability assessment.