How to Conduct AI Vulnerability Research in 2026 | AI LLM Hacking Course Day 43 of 90

In Day 42 — AI Social Engineering, we looked at how attackers use AI to make social engineering more convincing and effective. Today, in Day 43, I’m shifting our focus to independent AI vulnerability research. I’ll show you how to form testable hypotheses, investigate unexpected model behaviour, validate potential vulnerabilities, and report your findings responsibly rather than simply relying on techniques researchers already know.

In Day 44 — Secure AI Development, we’ll switch perspectives again. I’ll focus on how developers can apply these lessons to build AI systems that are more resilient to the attacks and vulnerabilities we’ve explored throughout the course.

Setting Up My AI Research Environment

Before I start investigating an AI vulnerability, I make sure I have a research environment where I can run experiments safely, repeat them, and compare the results. I don’t want my tests interfering with a production system or crossing boundaries I haven’t been authorised to test. For most of the exercises in this lesson, I’ll focus on two practical options: running an open-source model locally or using a dedicated API account for authorised research.

Let’s start with Ollama. When I run a local model such as Llama, Mistral, or Phi, I can experiment with prompts, compare responses, and repeat tests without paying for each API request. I’m also not consuming a third-party service’s API quota. However, there’s an important limitation I always keep in mind: a vulnerability I discover in a local model doesn’t automatically exist in a commercial model. Different model versions, system prompts, safety mechanisms, and application-level controls can produce very different results. Local testing is an excellent starting point, but I treat every finding according to the environment in which I actually reproduced it.

The second option is a dedicated API account used exclusively for research. This can help me investigate the behaviour of a specific hosted model, provided the provider permits the tests I’m conducting. I review the applicable usage policies and security research rules before starting, and I never assume that having an API key gives me unrestricted permission to test every feature or attack scenario. If a test requires additional authorisation, I obtain it first.

Whichever option I choose, I record the details that could affect my results: the model name and exact version, the date of testing, the system prompt configuration, relevant parameters, and any wrappers, plugins, or middleware involved. I also save the test inputs and outputs, removing secrets and personal information where necessary. Why am I so particular about this? Because a result that I can’t reproduce or explain is much harder to validate as a genuine security finding. Good research isn’t just about discovering unexpected behaviour; it’s about producing evidence that another researcher can independently verify.

Hypothesis Formation — How I Develop a Good Research Question

When I investigate an AI system, I try not to jump straight into testing random prompts. I’ve found that the most useful starting point is a clear hypothesis: an idea about how the system might behave under specific conditions that I can test and either support or reject with evidence. Without that starting point, it’s easy to spend hours collecting unusual responses without discovering whether any of them represent a genuine security issue.

Let me give you an example. Suppose I suspect that an AI model might reveal parts of its system prompt. Simply saying, “The model might leak its instructions,” isn’t a strong research hypothesis. It’s too broad to guide a meaningful experiment. I’d make the question more precise by identifying the conditions I’m interested in, the behaviour I expect to observe, and the evidence that would support my prediction. For instance, I could investigate whether changing the format of an otherwise equivalent, harmless request causes the model to disclose information it would normally keep private. I’d then compare the responses against a baseline and repeat the experiment to check whether the behaviour is consistent.

Notice what I’m doing here: I’m not assuming the vulnerability exists before I start testing. I’m making a prediction that could turn out to be wrong. That’s an important habit to develop. If the results don’t support my hypothesis, I record that outcome rather than changing the story to fit the evidence. If the behaviour does occur, I investigate whether it has a meaningful security impact and whether the result can be reproduced.

In my experience, useful research questions often come from three places. The first is unexpected behaviour. Perhaps a model produces a different response from the one I expected during a routine test. Rather than immediately calling it a vulnerability, I ask what caused the difference and design a controlled experiment to investigate it.

The second source is architectural reasoning. Here, I think about how the system is designed, what assumptions its components make, and where those assumptions might break down. For example, I might investigate how a model and its surrounding application distinguish trusted instructions from untrusted user content. This approach helps me ask questions about weaknesses that existing test cases may not cover.

The third source is ideas borrowed from other security disciplines. When I learned about SQL injection, one important lesson was that problems can arise when an application fails to separate instructions from data. I can use that general principle to ask questions about AI applications too: how does the system distinguish instructions from content, and what happens when that separation fails? The underlying technologies are different, so I don’t assume the same attack will work. Instead, I use the analogy to develop a new, testable question.

All three approaches are valuable. Unexpected behaviour gives me clues, architectural reasoning helps me explore possible weaknesses, and lessons from other security fields give me fresh ways to think about the problem. I don’t assume one approach will always produce better discoveries. What matters is whether I can turn an idea into a controlled experiment, collect reliable evidence, and demonstrate a security impact that stands up to scrutiny.

The Reproducibility Standard — How I Validate a Finding

One lesson I’ve learned about AI security research is that a surprising response doesn’t automatically mean I’ve discovered a vulnerability. Large language models can produce different answers to similar inputs, and their behaviour may depend on sampling settings, conversation history, system instructions, and the surrounding application. Sometimes, what looks like a security weakness on the first attempt simply doesn’t happen again. That’s why I never rely on a single response as proof.

When I find something interesting, I try to reproduce it under controlled conditions. I keep the model version and configuration consistent, record the original input and output, and repeat the experiment. Once I’ve established a baseline, I test carefully varied phrasings to see whether the behaviour persists or depends on one unusually specific prompt. I also check whether changes in the conversation context or model settings affect the result.

For the exercises in this lesson, I’ll use a practical starting benchmark: five successful demonstrations across at least three different input phrasings. This isn’t a universal industry standard, and passing it doesn’t automatically confirm a vulnerability. It’s a useful way to reduce the chance that I’m mistaking a one-off response for a repeatable pattern. Depending on the finding, I may need more trials, a different test design, or a clearer definition of what counts as success.

I pay particular attention to whether the result depends on one exact prompt. If I can reproduce the behaviour only with a single carefully constructed input, I document that limitation rather than claiming the issue is broadly exploitable. The finding might still be security-relevant, but I need more evidence before drawing conclusions about its practical impact. Likewise, a result that occurs at temperature zero isn’t automatically low severity; temperature affects output variability, while severity depends on factors such as the vulnerability’s impact, prerequisites, and realistic exploitability.

Finally, I keep a record of unsuccessful attempts as well as successful ones. That helps me understand the limits of the behaviour, prevents me from cherry-picking results, and makes it easier for another researcher to repeat my work. My goal is to produce a finding that can withstand independent verification, not just one impressive screenshot.

Documenting Findings — The Minimum Reproducible Example

When I discover something unusual during AI security testing, I don’t immediately write it up as a vulnerability. First, I ask myself a simple question: could another researcher reproduce the same behaviour using the evidence I’m about to provide? If the answer is no, I know my documentation needs more work. A screenshot of an unexpected model response might be interesting, but on its own, it rarely explains what happened, why it happened, or whether the behaviour represents a genuine security weakness.

That’s where the minimum reproducible example, often called an MRE, becomes useful. My aim is to create the smallest, clearest test case that demonstrates the behaviour without introducing unnecessary variables. I want another researcher to understand the setup, follow the steps, observe the result, and compare it with the expected behaviour. I don’t want them guessing which model I used, what instructions were active, or whether an earlier conversation influenced the output.

What I Include in a Minimum Reproducible Example

I start by recording the environment. For an AI vulnerability, this usually means the model name and version, the date of testing, the relevant configuration, and whether I tested a local model or an authorised hosted API. If the application uses a system prompt, retrieval pipeline, external tools, or middleware that affects the result, I document the relevant components too. I never assume that two systems using the same underlying model will behave identically.

Next, I record the test itself. I preserve the relevant input, the model’s observed output, the expected behaviour, and the steps needed to reproduce the result. I also note important settings, such as temperature, where applicable. If the result depends on conversation history, I document the necessary context using a minimal, sanitised example. This helps distinguish a behaviour caused by the test input from one caused by earlier instructions or application state.

Finally, I explain the security significance. An unexpected answer isn’t necessarily a vulnerability. I need to establish what security property is affected, what an attacker would need to do, and what realistic consequences could follow. If I can’t demonstrate meaningful impact, I describe the result as an observation or a potential issue rather than overstating my conclusion.

My Practical Documentation Workflow

• Step 1 — Record the hypothesis: Write down what behaviour I expected to observe and why it could matter for security.
• Step 2 — Establish the baseline: Record how the model behaves under ordinary conditions before introducing the test variation.
• Step 3 — Capture the minimum test case: Preserve the smallest authorised input and context that reproduce the behaviour.
• Step 4 — Record the result: Include the observed output, timestamp, model version, relevant settings, and whether the test succeeded.
• Step 5 — Repeat and compare: Run controlled repetitions and record failures as well as successes. Don’t select only the results that support the hypothesis.
• Step 6 — Explain the impact: Describe the affected security boundary, required conditions, potential consequences, and limitations of the evidence.
• Step 7 — Prepare the report: Remove unnecessary sensitive information and organise the evidence so an authorised recipient can validate the finding safely.

A Simple Example: Documenting Unexpected Model Behaviour

Imagine I’m testing a local language model and notice that it behaves inconsistently when asked to follow instructions embedded in an untrusted document. Rather than simply recording that the model followed an unexpected instruction once, I create a small test using synthetic content and a harmless action. I compare the result with a control case, repeat the experiment, and record the model configuration and surrounding application behaviour.

I then document whether the model followed the embedded instruction, whether the application had any controls to prevent it, and what the consequence would be if the same behaviour occurred in a real deployment. If I tested only the model in isolation, I make that limitation explicit. I don’t claim that a production application is vulnerable unless I have evidence supporting that conclusion in the relevant environment.

This distinction matters because the security risk often depends on the complete application, not just the language model. An unexpected response in a local experiment can help me develop a hypothesis, but demonstrating a vulnerability in a deployed product may require additional authorised validation of its integrations, permissions, and security controls.

Minimum Reproducible Example — Report Template

When I’m ready to share a finding, I use a structure like the one below. It keeps the report concise while preserving the information another researcher needs to investigate the issue.

• Title: A concise description of the suspected vulnerability.
• Environment: Model or application, version, test date, and relevant configuration.
• Prerequisites: The access, permissions, and conditions required for the test.
• Steps to reproduce: A minimal, numbered sequence of safe and authorised actions.
• Expected result: The behaviour I believe should occur.
• Observed result: What happened during the experiment, supported by relevant evidence.
• Reproducibility: The number of trials, variations tested, and any inconsistent results.
• Security impact: The affected security property and realistic consequences.
• Limitations: What I have not tested, what remains uncertain, and any environmental dependencies.
• Suggested mitigation: Practical defensive measures for the receiving team to evaluate.

Common Documentation Mistakes I Avoid

One mistake is providing only a screenshot without the context needed to reproduce the result. Another is omitting failed attempts, which can make a finding appear more reliable than it really is. I also avoid confusing model behaviour with application behaviour, claiming severity without evidence, or including real credentials and confidential information in a proof of concept.

I keep the report focused on the evidence and avoid adding complicated steps that aren’t necessary to demonstrate the issue. If a finding requires several conditions to work, I document each one. If I can’t reproduce it consistently, I say so. Honest limitations make a report more useful, not less valuable.

My final test is straightforward: could another authorised researcher follow my documentation, understand the result, and independently assess its security impact? If not, I improve the report before submitting it. That discipline makes it easier for security teams to investigate findings, compare results, and decide on appropriate remediation.

For the next step, read Documenting Findings — The Minimum Reproducible Example and use the principles above to improve your own research notes. Before publishing or sharing a proof of concept, make sure the affected system’s disclosure policy permits the proposed testing and publication.

Responsible Disclosure for AI Vulnerabilities — How I Report My Findings

Once I’ve reproduced a potential AI vulnerability and gathered enough evidence to explain its security impact, I move on to the next important step: reporting it responsibly. I treat this part of the process just as seriously as the technical research. Discovering a weakness is valuable, but how I communicate it can make a real difference to the people responsible for fixing it and to the users who might be affected.

I generally follow the same coordinated disclosure principles used in traditional software security. I contact the affected provider privately, explain the issue, include enough evidence to reproduce it safely, and give the organisation a reasonable opportunity to investigate and respond. A 90-day disclosure window is commonly used as a starting point in parts of the security research community, but I don’t treat it as a universal rule. The appropriate timeline depends on the provider’s published policy, the severity of the issue, the complexity of the fix, and any applicable disclosure commitments.

AI vulnerabilities can make remediation more complicated than I might expect. With a conventional software flaw, a developer may be able to correct a specific coding error and release a patch. With prompt injection, for example, the weakness may arise from how an application handles instructions, retrieves external content, grants tools access, or processes model outputs. Changing the system prompt alone may not eliminate the underlying risk. Effective mitigation could require changes to application architecture, access controls, input handling, output validation, monitoring, or several of these together.

When I prepare a report, I make the information as clear and useful as possible. I include a concise summary, the affected model or application, the prerequisites, reproducible steps, expected versus observed behaviour, evidence of security impact, and any limitations I’ve identified. I avoid including real user data, unnecessary secrets, or destructive proof-of-concept actions. I also explain what I have and haven’t tested so the receiving team can assess the finding accurately.

Before contacting a provider, I check its official security page, vulnerability disclosure policy, or bug bounty programme. For example, I would review OpenAI’s security information or Anthropic’s responsible disclosure guidance, where applicable, rather than assuming that a particular email address or programme accepts my type of report. I also verify the current scope and submission requirements because policies can change.

After submitting the report, I keep a record of when I contacted the provider, what I shared, and any acknowledgement or follow-up I received. If I don’t hear back within the timeframe specified in the policy, I follow up through the appropriate channel. If the organisation remains unresponsive, I may seek advice from a recognised vulnerability coordination organisation, such as a relevant CERT, before considering any public disclosure.

Finally, I don’t assume that the passage of 90 days automatically makes public disclosure appropriate. I consider the risk to users, the provider’s response, the availability of effective mitigations, and the potential harm of releasing technical details. My aim is to help the issue get addressed while allowing the wider security community to learn from the research without unnecessarily exposing people to harm.

CVE Classification and MITRE ATLAS — How I Classify AI Vulnerabilities

Once I’ve reproduced a potential AI vulnerability and documented the evidence, I need to answer another important question: how should I classify the finding? I don’t want to give every unexpected model response a vulnerability label or assume that every AI security issue qualifies for a CVE. My first job is to understand what has actually gone wrong, which component is affected, and whether the behaviour creates a demonstrable security risk.

This is where two resources become useful: the Common Vulnerabilities and Exposures (CVE) programme and MITRE ATLAS. They serve different purposes. CVE provides identifiers for publicly disclosed cybersecurity vulnerabilities that meet the programme’s requirements. MITRE ATLAS provides a knowledge base of adversary tactics and techniques associated with attacks against AI-enabled systems. I use them together when appropriate, but I don’t treat them as interchangeable classification systems.

What Makes an AI Finding a Potential CVE?

When I evaluate a finding for possible CVE assignment, I start by examining the underlying security weakness rather than focusing only on the AI terminology. Is there a vulnerability in the application’s access controls, an unsafe integration with an external tool, an exposed secret, or an input-handling flaw that creates a security boundary violation? These details help me determine whether the finding represents a vulnerability that may fit the CVE programme’s scope.

For example, imagine an AI assistant connected to an internal document system. During authorised testing, I discover that a user can retrieve documents they are not permitted to access because the application fails to enforce document-level permissions. The presence of an LLM makes this an AI-related security issue, but the underlying weakness may be broken access control in the application. I would document the access-control failure, its prerequisites, and its impact instead of automatically labelling it a new model vulnerability.

Now consider a model that produces an unexpected response to a particular prompt. That observation alone doesn’t establish a CVE-worthy vulnerability. I need to determine whether it violates a meaningful security boundary, whether the behaviour is already known, whether the impact can be demonstrated, and whether the affected product or component falls within the relevant programme’s criteria.

I also check whether the issue has already been reported or assigned an identifier. If a vulnerability is eligible for CVE assignment, the appropriate vendor, CNA (CVE Numbering Authority), or other authorised participant can handle the assignment according to the applicable process. Researchers should not invent CVE identifiers or assume that submitting a report guarantees an assignment.

Where MITRE ATLAS Fits In

I use MITRE ATLAS to understand how an AI-related finding fits into the broader threat landscape. ATLAS stands for Adversarial Threat Landscape for Artificial-Intelligence Systems. It documents adversary tactics, techniques, and case studies involving attacks against AI-enabled systems, helping security teams reason about how attacks may be carried out and where defensive controls are needed.

Suppose my research involves prompt injection in an AI application that retrieves external documents. I can consult ATLAS to identify relevant documented techniques and understand how the observed behaviour relates to known adversarial patterns. I then compare those techniques with the actual evidence from my experiment rather than choosing a label simply because it sounds relevant.

This distinction matters: mapping a finding to an ATLAS technique does not prove that the system is vulnerable, and an ATLAS technique is not a CVE identifier. ATLAS helps me describe the attack behaviour and threat context; my technical evidence establishes whether a vulnerability exists and what impact it has.

My Practical Classification Workflow

• Step 1 — Define the weakness: Describe the exact behaviour and identify the affected component, such as the model, application, retrieval system, tool integration, or access-control layer.
• Step 2 — Validate the impact: Establish what security property is affected and what an attacker could realistically achieve under the documented conditions.
• Step 3 — Check existing knowledge: Search vendor advisories, relevant vulnerability databases, and published research to see whether the behaviour is already documented.
• Step 4 — Review CVE eligibility: Check the current CVE programme requirements and consult the affected vendor or an appropriate CNA when necessary.
• Step 5 — Map relevant ATLAS techniques: Use the official MITRE ATLAS knowledge base to identify techniques that accurately describe the observed adversarial behaviour.
• Step 6 — Record the evidence: Document the reproduction steps, affected versions, prerequisites, limitations, and supporting observations.
• Step 7 — Report responsibly: Submit the finding through the appropriate security disclosure channel and coordinate any publication according to the applicable policy.

Example: Classifying a Prompt Injection Finding

Imagine I’ve tested an AI application that summarises documents retrieved from an authorised test repository. One document contains untrusted instructions, and the application follows those instructions in a way that conflicts with its intended behaviour.

I would first reproduce the behaviour and establish whether the application actually crosses a security boundary. Does it merely generate an unexpected summary, or can the behaviour cause a meaningful security consequence, such as exposing information the user is not authorised to access? I would record the answer rather than assume the worst-case scenario.

Next, I would examine the application’s trust boundaries and existing safeguards, then consult MITRE ATLAS for relevant documented techniques. If the issue exposes a separate application weakness, such as missing authorisation checks, I would document that weakness explicitly. I would then discuss CVE eligibility with the relevant vendor or authorised numbering authority if the finding appears to meet the applicable requirements.

The outcome might be a reportable application vulnerability, a useful security observation, a known attack pattern, or a combination of these. The classification must follow the evidence. I don’t force every finding into a CVE or ATLAS category just to make the report appear more significant.

My Classification Checklist

• Have I demonstrated a security weakness rather than merely an unexpected output?
• Can I explain the impact, prerequisites, and affected component?
• Have I checked for existing reports, advisories, or known techniques?
• Have I verified that any proposed ATLAS mapping accurately describes the observed behaviour?
• Have I checked the current CVE eligibility and assignment process?
• Have I separated confirmed facts from assumptions and potential consequences?
• Have I prepared a reproducible report and followed the appropriate disclosure process?

My takeaway is simple: classification should make a security finding easier to understand, investigate, and communicate. CVE helps identify qualifying vulnerabilities, while MITRE ATLAS helps explain adversarial techniques targeting AI systems. When I use both carefully, I can place my research in context without overstating what the evidence proves.