How to Respond to an AI Incident in 2026 | AI LLM Hacking Course Day 40 of 90

In Day 39, we looked at the governance side of AI security — including what your incident management process needs to look like before anything goes wrong. Today, in Day 40, I want to make that practical. If an AI security incident actually happens, what do I look at first, how do I contain it, what evidence do I preserve, and how do I recover without making things worse? That’s the operational incident response process we’ll work through here. Then, in Day 41, we turn the perspective around again and look at advanced red-team techniques — combining the skills from Days 1–40 to simulate more sophisticated, multi-stage attacks.

AI-Specific Incident Detection Signals

When I investigate a traditional breach, I usually have familiar things to look for: unusual network connections, unexpected file changes, suspicious authentication, new processes, privilege escalation, and other endpoint or network indicators. With an AI incident, I still check those things — but I don’t stop there. The infrastructure can look completely normal while the AI itself is behaving in a way it shouldn’t.

That’s the mindset I want you to develop here. Instead of asking only, “Is the server compromised?”, I also ask, “Is the AI behaving differently from its expected baseline?” I look for responses that contain information the user never requested, instructions that appear to have come from somewhere else, tool calls that don’t make sense for the request, sudden changes in how the model follows instructions, and unusual query patterns that look more like systematic probing than normal use.

For example, if a customer asks about a product and the model suddenly starts recommending a competitor that wasn’t mentioned anywhere in the conversation, that’s worth investigating. If an agent calls a sensitive tool even though nothing in the user’s request should have triggered it, that’s another signal. If the model suddenly starts revealing system-prompt content or returning credential-like strings, I treat that very differently from an ordinary bad response.

None of these signals should automatically be treated as proof of compromise. That’s important. A keyword match can produce false positives, and an unusual response can simply be a legitimate edge case. What I’m looking for is evidence that becomes meaningful when I correlate it with the rest of the AI activity.

This is where the production monitoring layer from Day 35 becomes useful. The logging and telemetry are already there; now I need detection rules that understand AI behaviour. Here are some practical starting rules I can build into that monitoring layer:

One important point: I don’t rely on any single rule to declare an incident. A prompt containing “ignore previous instructions” might simply be a security researcher testing the application. A competitor mention might be legitimate. A long response might be expected for a particular workflow. The real detection value comes from correlation — what the user asked, what data was retrieved, what tools were called, what the model produced, and how that behaviour compares with the established baseline.

That shift — from looking only for compromised infrastructure to also looking for compromised behaviour — is one of the biggest differences between traditional incident response and AI incident response.

Incident Classification and Response Paths

Once I have enough evidence to believe something unusual is happening, I don’t immediately start changing things. First, I need to understand what kind of incident I’m dealing with. The classification matters because a prompt-injection attempt from a single user needs a very different response from a compromised RAG knowledge base that has already affected thousands of conversations.

I normally classify the incident along three dimensions: what was affected, how far the attacker got, and whether the attack is still active. That gives me a much clearer response path than simply labelling everything an “AI security incident.”

1. Attack Attempt — No Confirmed Impact

This is where I see evidence of an attack attempt, but I can’t yet demonstrate that the AI system was actually compromised. A user may submit a prompt-injection payload, probe for the system prompt, attempt to bypass a safety control, or send a sequence of jailbreak prompts.

My response here is usually investigation rather than emergency containment. I preserve the relevant prompts and responses, identify the account or session, check whether any tools were invoked, and look for similar activity from the same source. If the attempt was blocked and nothing else was affected, I don’t want to create unnecessary operational disruption.

2. Model Behaviour Anomaly — Possible Compromise

The next level is when the AI starts behaving differently from its established baseline. Maybe it suddenly reveals information it normally refuses to provide, follows instructions from retrieved content, calls tools in an unexpected sequence, or produces outputs that don’t match the application’s intended behaviour.

At this point, I treat the behaviour as a security signal rather than assuming it’s simply a model-quality problem. I compare the affected interactions with normal traffic, inspect the retrieved context, review recent changes, and check whether the behaviour can be reproduced.

3. Data or RAG Compromise

This is particularly important for production RAG systems. If malicious or manipulated content has entered the knowledge base, the attack can affect many users without the attacker ever touching the model infrastructure.

My response changes here. I identify the affected documents, ingestion job, source system, embeddings and retrieval path. I may temporarily remove the affected content from retrieval, quarantine the source, or disable the affected ingestion pipeline while preserving the original evidence for forensic analysis.

4. Tool or Agent Compromise

If an AI agent has access to email, databases, APIs, cloud resources or other tools, I treat unexpected tool activity much more seriously. The question is no longer just whether the model produced a bad answer. I need to determine whether the model actually caused an external action.

This is where I move quickly to contain the affected capability. Depending on the architecture, that might mean disabling a particular tool, revoking temporary credentials, restricting the agent’s permissions, or moving the workflow into a human-approval state.

5. Confirmed Security Incident

I classify an incident as confirmed when I have sufficient evidence that an attacker or malicious input caused an unauthorized change, disclosure, action, or security-control bypass. At that point, I’m no longer treating the event as an anomaly to investigate at my leisure. I’m in incident-response mode.

The priorities become straightforward: contain the attack, preserve evidence, determine scope, eradicate the cause, and recover safely. I also start recording decisions and timestamps because the investigation itself becomes part of the forensic record.

6. Business-Impact Incident

Sometimes the technical compromise is relatively small but the business impact is significant. An AI system might give incorrect financial advice, expose customer information, make unauthorized changes, or systematically provide misleading recommendations.

In those cases, I don’t let the technical team make the entire decision in isolation. Security, engineering, legal, compliance, privacy and the affected business owner may all need to be involved. The response path depends not only on how the AI was compromised, but on what the compromise actually did.

Choosing the Response Path

Once I’ve classified the incident, I choose the narrowest response that contains the risk without destroying evidence or unnecessarily taking the whole system offline.

Prompt-injection attempt: preserve the interaction, identify the source, investigate repetition and strengthen the relevant control.

Behaviour anomaly: compare against the baseline, trace the prompt and context chain, and determine whether the behaviour is reproducible.

RAG poisoning: quarantine the affected data and ingestion path, identify the scope of exposure, and rebuild or validate the affected retrieval layer.

Unexpected tool activity: restrict the affected tool or capability, review credentials and permissions, and determine exactly what actions were executed.

Confirmed compromise: activate the formal incident-response process, preserve evidence, contain the affected components, investigate the full attack path, and coordinate recovery.

The important thing is that I don’t use “take the AI offline” as my automatic answer. Sometimes that’s appropriate. Sometimes it creates more damage than the incident itself. If I can isolate a poisoned knowledge source, disable one tool, or place an agent behind human approval, I may be able to contain the incident while keeping the rest of the service operational.

That decision — what to isolate, what to leave running, and what evidence to preserve before making the change — is where AI incident response becomes a real operational discipline rather than just a checklist.

Graduated Containment Options

When I confirm an AI security incident, my first instinct isn’t to shut down the entire AI system. Sometimes that’s necessary, but taking everything offline immediately can interrupt legitimate users, destroy useful runtime evidence, and make it harder to understand what the attacker was actually doing.

Instead, I use graduated containment. I start with the smallest action that can reliably stop the harmful behaviour, then escalate if I can’t control the incident at that level. The goal is simple: stop the attack without creating more damage than I need to.

Level 1 — Contain the Session or User

If the activity is isolated to one account, API key, conversation, or session, that’s where I start. I may terminate the session, temporarily restrict the account, revoke the affected API token, or prevent that identity from invoking sensitive AI capabilities.

Before I do that, I preserve what I can: the original prompt, complete conversation history, model response, timestamps, retrieved context, tool calls, session identifiers, authentication events, and relevant request metadata. I don’t want containment to erase the trail I’m about to investigate.

Level 2 — Disable the Affected Capability

Sometimes the user isn’t the real problem. The dangerous part may be one capability available to the model. If an agent is making suspicious database queries, for example, I may disable that database tool while leaving ordinary conversational functionality available.

The same principle applies to email actions, code execution, file access, web retrieval, payment functions, administrative APIs, and other privileged tools. If I can remove the capability that turns a malicious prompt into a real-world action, I’ve reduced the immediate risk without necessarily shutting down the whole application.

Level 3 — Isolate the RAG or Data Source

For a suspected RAG poisoning incident, I don’t immediately assume the model itself needs to be replaced. I trace which documents were retrieved, where they came from, when they were ingested, and what other queries may have retrieved the same content.

If I identify a suspicious source, I can quarantine that document, collection, connector, index partition, or ingestion pipeline. I then check whether related content entered through the same path. This is especially important because deleting one visible malicious document doesn’t tell me whether ten more were ingested in the same batch.

Level 4 — Restrict Model or Agent Permissions

If I can’t confidently identify one affected tool, I reduce what the AI is allowed to do. An autonomous agent might temporarily become read-only. High-impact actions might require human approval. External API access might be restricted. Service-account permissions might be reduced to the minimum required for investigation.

I find this particularly useful when I’m still determining scope. I may not yet understand the full attack path, but I can reduce the blast radius while the investigation continues.

Level 5 — Isolate the AI Component

If malicious behaviour continues across multiple users, data sources, or tools, I move containment outward. I may isolate the affected model endpoint, agent, orchestration service, vector store, plugin, connector, or application instance from the rest of the environment.

At this stage, I also become much more cautious about credentials. If the affected component had access to secrets or privileged tokens, I identify exactly which credentials were reachable and rotate or revoke those that I have reason to consider exposed. I don’t blindly rotate everything before preserving the relevant evidence and understanding the dependencies.

Level 6 — Full Service Shutdown

This is the option I reserve for incidents I can’t safely contain at a lower level. If sensitive data is actively leaking, unauthorized actions are continuing, the integrity of multiple AI components is uncertain, or I can’t determine the boundaries of the compromise, taking the affected service offline may be the safest decision.

At that point, availability becomes secondary to limiting further harm. But even here, I want the shutdown to be deliberate. Where practical, I preserve volatile evidence, active sessions, relevant logs, configuration state, model and prompt versions, retrieval records, and tool-execution history before changing the environment.

How I Decide How Far to Escalate

I base the containment level on impact, scope, confidence, and ongoing risk. Is customer data leaving the environment? Can the model execute external actions? Is the behaviour limited to one session or appearing across many users? Do I know where the malicious instruction entered the pipeline? And most importantly, is the harmful activity still happening?

If I can confidently isolate one poisoned document and stop the behaviour, I probably don’t need to shut down the entire AI platform. If I can’t tell whether the model, RAG layer, tools, credentials, or orchestration environment can still be trusted, I escalate.

One thing I don’t want you to confuse is containment with recovery. If I disable a compromised tool and the suspicious behaviour stops, I’ve contained the immediate threat. I haven’t necessarily fixed the vulnerability, removed every malicious artifact, determined the full impact, or proved that the environment is safe to return to normal operation.

That’s why my containment question is deliberately narrow: “What can I isolate right now to stop further harm while preserving enough evidence to understand what happened?” Once I have that under control, I can move into deeper forensic investigation, eradication, and recovery.

AI Incident Forensics

Once I’ve contained the immediate problem, I don’t start deleting things and rebuilding the environment. First, I want to understand what actually happened. That’s where AI incident forensics becomes different from the traditional forensic process I’m used to.

In a conventional breach, I might spend a lot of time looking at disk artifacts, process execution, authentication events, network connections, and endpoint telemetry. I still want all of that for an AI incident, but it may not tell me why the AI started behaving differently. The most important evidence can be sitting inside the AI application’s own operational history.

So the first thing I want is the complete prompt-and-response trail for the incident window. I want to know what users asked, exactly what the model received, what it returned, when each interaction happened, and which session or request produced it. If the application uses RAG, I also want to know what documents were retrieved for each request. If the model can call tools, I want the corresponding tool-invocation records.

That last part is particularly important. A model response tells me what the AI said. A tool log can tell me what the AI actually did. If an agent sent an email, queried a database, modified a record, called an external API, or accessed a sensitive resource, I need evidence of that action independently of the model’s text response.

I also want to reconstruct the AI configuration as it existed during the incident. This means recording the model and deployment version, system-prompt version, safety configuration, tool definitions, application configuration, and relevant RAG state. A current configuration isn’t enough — I need to know what the system looked like when the suspicious behaviour occurred.

For a RAG incident, I go one step further. I want the document that was retrieved, its source, ingestion timestamp, document version or hash where available, embedding/index information, and the ingestion job that introduced it. I’m trying to answer a very specific question: did the attacker manipulate the AI at inference time, or had malicious content already entered the system before the incident began?

The Evidence I Want to Preserve

For a serious AI incident, I try to preserve the evidence before making changes that could overwrite it. At minimum, I’m looking for:

• Prompt and response logs — the complete interaction history for the affected period.
• Session and request identifiers — so I can connect individual interactions across services.
• Retrieved context — the documents, chunks, metadata, and sources supplied to the model.
• Tool invocation records — what tools were called, with what parameters, and what they returned.
• Model and prompt versions — exactly what configuration produced the responses.
• RAG and data-source state — what content was available to retrieval at the time.
• Authentication and access logs — who accessed the AI components and data pipelines.
• Application and infrastructure logs — supporting evidence that may establish how the attack entered or moved through the environment.

I don’t necessarily need every one of these to prove every incident. The point is to preserve enough evidence to reconstruct the attack path and establish scope before the normal operation of the system changes the evidence.

Why Configuration History Matters

One of the easiest mistakes to make is investigating the system as it exists today rather than the system that existed during the incident.

Imagine I discover suspicious responses on Monday and deploy a new system prompt on Tuesday. If I only examine Tuesday’s configuration, I may never understand why Monday’s responses were possible. The same problem occurs if the RAG index is rebuilt, a malicious document is deleted, or the model is upgraded before I capture the relevant state.

That’s why versioning matters. I want snapshots or immutable records of the configurations that were active during the incident window. Without them, I can identify that something went wrong without being able to confidently explain why.

Building the Incident Timeline

Once I’ve collected the evidence, I put it into a timeline. I start before the first known anomaly and work forward:

The timeline lets me connect events that may look unrelated when viewed separately. A suspicious document four days earlier, a retrieval event this morning, an unusual response ten minutes later, and an unexpected tool call can become one coherent attack chain once I put them in chronological order.

And this is the point of AI forensics: I’m not just trying to prove that the AI produced a bad answer. I’m trying to reconstruct the chain of events that caused it — what entered the system, what the model received, what it produced, what actions followed, who was affected, and whether anything remains compromised.

Without that reconstruction, recovery is mostly guesswork. With it, I can move into eradication with a much better idea of what actually needs to be removed.

Eradication and Recovery

Once I know what happened and I’ve contained the incident, I can start removing the cause. This is where I have to be careful with AI systems because not every AI incident requires replacing the model. Before I rebuild anything, I want to answer one question: what was actually compromised?

If the incident was caused by an inference-time attack — for example, a prompt injection that exploited the way the application handled instructions — the model itself may be perfectly clean. In that situation, replacing the model doesn’t solve the underlying problem. I need to fix the configuration or application layer that allowed the attack to work.

That might mean correcting the system prompt, fixing the instruction-handling logic, tightening the affected input or retrieval path, removing poisoned RAG content, and rotating credentials if the attacker could have accessed them. I then test the corrected system against the original attack path to make sure the behaviour is actually gone.

The distinction becomes much more serious if I find evidence that the model itself has been compromised — for example, through poisoned fine-tuning data or malicious model weights. At that point, changing the system prompt isn’t enough. I need to replace the affected model with a version whose integrity I can verify.

If I have a known-good previous model version, rolling back is usually the fastest recovery option. But I don’t treat an older version as automatically safe. I still need to establish where the compromise came from and whether the same poisoned dataset, training pipeline, dependency, or deployment process could compromise the replacement again.

I Verify Before I Restore

One of the biggest mistakes I can make during recovery is being in a hurry to declare the incident over. The service may be responding normally again, but that doesn’t prove the environment is clean.

Before I return the system to normal operation, I verify the model or configuration against a known-good state, validate the RAG data and ingestion pipeline, check relevant credentials, review tool permissions, and rerun the attack scenarios that originally triggered the incident.

I also want to know whether the fix works outside the exact example I investigated. If one malicious document caused the incident, removing that document isn’t enough. I test whether another document containing the same type of instruction can produce the same behaviour. If a jailbreak worked against one prompt, I test variations rather than assuming that changing one keyword solved the problem.

Recovery Is a Controlled Return

I prefer to bring an affected AI service back gradually when the architecture allows it. That might mean starting with a restricted deployment, read-only tools, reduced permissions, limited traffic, or additional monitoring before restoring full functionality.

During this period, I’m watching closely for the behaviour that originally triggered the incident. If the system starts showing the same anomaly again, I want to catch it before the entire production environment is exposed.

The Supply Chain Has to Be Part of the Fix

This is especially important when the compromise involves model weights or training data. I can’t consider eradication complete simply because I’ve replaced one bad file.

I need to examine the path that produced and delivered that file: training datasets, fine-tuning jobs, model repositories, CI/CD pipelines, dependencies, access controls, artifact storage, and deployment processes. If the same attacker-controlled input can enter again, the next model may be compromised before it ever reaches production.

So my recovery checklist looks something like this:

The rule I keep coming back to is simple: I don’t call an AI incident recovered just because the symptoms have disappeared. I call it recovered when I have reasonable evidence that the malicious artefact is gone, the entry point has been closed, the replacement components are trustworthy, and the same attack no longer produces the same result.

That’s the difference between restarting an AI service and actually recovering it.

The AI Incident Response Playbook

At this point, we’ve covered the individual pieces of AI incident response. Now I want to put them together into something I could actually use during an incident.

When an AI system starts behaving strangely, I don’t want to rely on memory or improvise the response while everyone is under pressure. I want a simple sequence that tells me what to establish first, what evidence to preserve, what I can safely change, and when it’s appropriate to bring the system back online.

The playbook below is the sequence I use as a mental model. The exact technical steps will depend on the architecture, but the order matters.

Phase 1 — Detect and Triage

First, I establish whether I’m looking at a genuine security incident or simply an application or model-quality problem. I start with the suspicious behaviour and work backwards.

• Identify what changed and when it first appeared.
• Compare the behaviour with the established baseline.
• Identify affected users, sessions, models, tools and data sources.
• Check whether the behaviour is still occurring.
• Assign an initial severity based on actual impact and exposure.

I don’t need the complete root cause at this stage. I need enough confidence to decide whether I have to contain something immediately.

Phase 2 — Preserve Evidence

Before making changes, I preserve the evidence that could disappear when the system changes. That includes prompt and response logs, retrieved context, tool calls, authentication records, configuration versions, model versions, RAG state, and relevant infrastructure telemetry.

I also establish the incident timeline. When did the suspicious content enter? When was it first retrieved? When did the behaviour begin? Which users were affected? What actions did the AI take? What happened immediately before containment?

If I can’t reconstruct the timeline later, I’ve probably lost something important during the response.

Phase 3 — Contain

Next, I stop the active risk. I start as narrowly as possible and escalate when necessary.

• Restrict the affected user, session or API key.
• Disable the affected tool or capability.
• Quarantine suspicious RAG content or an ingestion source.
• Reduce agent permissions or require human approval.
• Isolate the affected AI component.
• Take the service offline if the risk cannot be safely contained another way.

The important thing is that I don’t confuse stopping the behaviour with fixing the incident. Containment buys me time. The investigation still has to explain what happened and whether anything else was affected.

Phase 4 — Investigate

Once the immediate risk is controlled, I follow the attack through the AI pipeline. I want to know where the malicious instruction or data entered, how it reached the model, what the model did with it, and whether that behaviour produced any external impact.

For a RAG incident, that means tracing the document through ingestion, indexing and retrieval. For an agent incident, I trace the tool calls and permissions. For a model compromise, I investigate the model artifact, training or fine-tuning pipeline and deployment history.

At this stage, I’m also trying to establish scope. One affected conversation is very different from six hours of affected customer interactions, and both are different from a compromised model being deployed across the entire organization.

Phase 5 — Eradicate

Now I remove the actual cause rather than just the visible symptom.

If it was an application or prompt-injection weakness, I fix that weakness. If malicious RAG content entered the system, I remove it and investigate the ingestion path. If credentials were exposed, I rotate them and review their permissions. If the model artifact itself was compromised, I replace it with a verified version and investigate the supply chain that produced it.

I also ask one uncomfortable question here: could the attacker do the same thing again? If the answer is yes, I haven’t finished eradication.

Phase 6 — Recover Carefully

I don’t immediately switch everything back to normal. Where the architecture allows it, I restore the service gradually and keep additional monitoring in place.

I validate the model and configuration, test the original attack path, verify the RAG sources, check tool permissions, and watch for the same behavioural indicators that triggered the incident.

If the system passes those checks, I can progressively restore normal functionality. If the suspicious behaviour returns, I stop the recovery and go back to investigation.

Phase 7 — Close the Incident

The incident isn’t finished when the application is back online. I still need to document what happened, what was affected, what evidence supports the conclusions, which controls failed, and what needs to change.

This is also where the governance work from Day 39 becomes practical again. The incident record, regulatory assessment, control changes, ownership and follow-up actions all need to feed back into the organization’s broader AI risk-management process.

That’s the playbook I want you to remember: detect, preserve, contain, investigate, eradicate, recover, and learn.

The order isn’t arbitrary. If I contain before preserving critical evidence, I may lose part of the attack trail. If I eradicate before understanding the root cause, I may remove the obvious malicious artefact while leaving the entry point open. And if I recover without testing the original attack path, I may simply put the same vulnerability back into production.

Good AI incident response isn’t about reacting quickly and changing everything. It’s about moving quickly without losing control of the investigation.