AI Gradual Escalation Lab: Small Steps Past the Guardrails

This article describes a cybersecurity lab that demonstrates 'gradual escalation' or 'boil the frog' attacks against AI language models, where attackers achieve disallowed outcomes through a sequence of small, individually acceptable requests rather than one direct request. The lab uses a system called BoilBot that tracks internal 'comfort' levels, showing how compliance drift—the model's difficulty refusing after a run of agreements—enables attackers to move the model past boundaries it would defend in isolation. The problem matters because message-by-message safety evaluation fails to capture session-level attacks, leaving guardrails vulnerable to patient, incremental exploitation.

• Compliance drift mechanism in BoilBotBoilBot tracks a 'comfort' metric that rises with each accepted message. This internal state means that after a sequence of agreements, the model finds it harder to refuse the next small step because doing so would feel inconsistent with its prior behavior. The lab demonstrates that no single step in the sequence looks alarming individually, but the accumulated trajectory moves the model past its original boundary.
• Attack requires small incremental stepsThe attack methodology starts with a clearly acceptable request, then escalates in small increments where each step is only slightly beyond the last. The attacker must keep each step individually reasonable so none triggers a refusal. If a step gets refused, the attacker backs off and opens the steps further. This patience is essential—one large leap will trip a refusal.
• Message-by-message evaluation misses trajectoryThe core vulnerability is that models evaluate each message largely on its own while relying on helpful-by-default training. This means a chain of agreements builds momentum; the model evaluates safety per message rather than across the session, allowing the trajectory itself to become the attack vector. A boundary that holds against one direct request often falls to a patient ramp of smaller requests.
• Session-level aggregation as primary defenseThe article identifies that guardrails must score the entire session, not just individual messages, aggregating risk across turns. Hard boundaries must be anchored independently so prior compliance cannot lower them. Additionally, rules should be reset or re-asserted periodically within long conversations, and escalation patterns showing steadily rising sensitivity should be flagged for review.
• Classification as OWASP LLM01 multi-turnThe article maps gradual escalation to OWASP LLM01 Prompt Injection, specifically in its multi-turn form. This connects the technique to the OWASP Top 10 for LLM Applications (2025), indicating it is a recognized and categorized attack vector that defenders should understand as part of broader LLM security.
• Why single-message classifiers failMulti-turn escalation and related many-shot techniques are documented ways to erode refusals, which is why single-message classifiers are insufficient according to the article. The model's evaluation logic in isolation cannot detect or prevent the cumulative effect of a carefully constructed sequence designed to shift internal state over multiple turns.

Bottom line: Gradual escalation attacks succeed because AI safety mechanisms that evaluate messages in isolation miss the session-level trajectory; defenders must aggregate risk across the entire conversation, anchor hard boundaries independently of prior compliance, and actively detect escalation patterns rather than relying on per-message guardrails alone. Testing a model's robustness requires probing sequences, not just single prompts, since boundaries that resist direct requests often fall to patient, in