How to Assess LLM Fine Tuning Security in 2026 – Dataset Poisoning, Training Attacks and Fine-Tune Vulnerabilities | AI LLM Hacking Course Day 38 of 90

In Day 37, I looked at the privacy attack surface of AI systems that handle personal data. Today, in Day 38, I’m moving one step earlier in the lifecycle — into the fine-tuning process itself. I want you to understand what can go wrong inside the training pipeline before the model ever handles a user’s request.

Then, in Day 39, we’ll step back and look at the bigger picture: AI governance and compliance security testing, including how to assess an organisation’s AI governance posture against frameworks such as the NIST AI RMF and the EU AI Act.

Fine-Tuning Dataset Audit

I treat a fine-tuning dataset as a software supply-chain artefact, not simply as a collection of training examples. By the time the dataset reaches the training job, it may have passed through document repositories, databases, annotation platforms, ETL jobs, preprocessing scripts, third-party datasets, and automated data-generation systems. Every one of those stages can introduce a security or integrity problem.

So when I audit a fine-tuning dataset, I start with a simple question: Where did every training example come from? I want to be able to trace the major data sources back to their origin and understand how they moved through the pipeline before becoming part of the training set.

That means creating a source inventory covering internal documents, customer interactions, support tickets, web-scraped content, third-party datasets, licensed data, human annotations, and synthetic examples generated by other models. For each source, I look at who owns it, who can modify it, how changes are recorded, how the data is ingested, and whether there is an integrity check between ingestion and the final training dataset.

I also want to know when the data changed. A dataset containing a suspicious cluster of examples is much easier to investigate when I can correlate those examples with a specific document upload, database change, annotation batch, or ingestion job. Without timestamps, versioning, and provenance information, determining where a poisoned example came from becomes significantly harder.

The most commonly underestimated poisoning surface is often the internal document repository. Security teams naturally treat external datasets as untrusted, but internal content can receive far less scrutiny simply because it was produced by employees or stored behind corporate authentication.

That assumption is dangerous. An authenticated user does not automatically mean a trusted data source. If an employee can modify documents that are automatically consumed by the fine-tuning pipeline, their write access may indirectly become training-data write access.

Imagine a pipeline that periodically collects internal product documentation and converts it into training examples. An authorised employee adds a document containing systematically misleading product comparisons. The document looks legitimate, passes through the normal ingestion process, and eventually becomes part of the fine-tuning dataset. There may be no exploit, malware, or compromised server involved. The security boundary failed because the pipeline trusted the source too much.

This is why I map data write access to training influence. For every important source, I ask: who can change it, can those changes reach the training pipeline automatically, and what controls exist between the change and the training run?

What I Check During the Audit

• Dataset provenance: Can important training examples be traced back to their original source?
• Source ownership: Is there a clearly identified owner for every data source?
• Write permissions: Who can add, modify, or delete content that eventually reaches the dataset?
• Version history: Are changes to source documents and datasets recorded and recoverable?
• Integrity controls: Are hashes, signed manifests, immutable versions, or equivalent controls used to detect unexpected changes?
• Preprocessing: Can transformation or cleaning scripts introduce or amplify unwanted patterns?
• Deduplication: Can repeated examples cause a small poisoned source to have disproportionate influence during training?
• Sampling: How are examples selected for the actual fine-tuning run?
• Human review: Are high-impact or suspicious examples reviewed before training?
• Dataset versioning: Can the exact dataset used for a particular model version be reconstructed?
• Training linkage: Can the organisation identify which dataset version produced which model checkpoint?
• Post-training validation: Does evaluation specifically test for unexpected behaviour introduced during fine-tuning?

I pay particular attention to the last three points. A dataset can be perfectly documented and still produce an unsafe model. The security team therefore needs to maintain a chain of evidence from source → dataset → training run → model version → evaluation results.

Look for Disproportionate Influence

Another useful question is whether a relatively small source can have an unusually large influence on the resulting model. A source containing thousands of examples may naturally have more influence than one containing a few examples, but that does not mean every source should be treated equally.

During an assessment, I look for unusual concentrations of examples, repeated patterns, unexpected duplication, sudden changes in topic or sentiment, and examples that appear shortly before a training run. These are not proof of poisoning on their own. They are investigation signals that tell me where to look more closely.

I also compare the dataset against earlier versions whenever possible. If a behaviour appears in the fine-tuned model but not in the base model, I want to know what changed in the training data between the two versions. A clean dataset diff can sometimes explain a behavioural change much faster than inspecting the model itself.

Deliberate Poisoning vs. Accidental Bias

One important distinction is that not every problematic training example represents an attack. A dataset can contain accidental bias, outdated information, poor labelling, duplicated content, or organisational assumptions that were never intentionally introduced to manipulate the model.

From a security perspective, however, the remediation question is often similar: Can an unauthorised or insufficiently controlled change to the training data materially alter model behaviour?

If the answer is yes, I treat that as a meaningful weakness in the fine-tuning security surface, regardless of whether the original cause was malicious or accidental.

The Audit Evidence I Want

A good dataset audit should produce evidence rather than simply a statement that the dataset was reviewed. I want to see the dataset manifest, source inventory, access-control records, version history, ingestion logs, transformation steps, dataset hashes or equivalent integrity records, training-run identifiers, and evaluation results.

The goal is straightforward: if someone asks six months later, “Why does this model behave differently from the previous version?”, the security team should be able to reconstruct what changed in the data, who could have changed it, which training run consumed it, and which evaluation detected or failed to detect the resulting behaviour.

That is the real purpose of a fine-tuning dataset audit. I’m not just checking whether the dataset contains bad examples. I’m checking whether the organisation has enough provenance, integrity, access control, and traceability to prevent, detect, and investigate changes that could influence model behaviour.

Safety Degradation Testing

One of the mistakes I see in fine-tuning security assessments is treating safety evaluation as something that happens only once. A model passes the safety checks before fine-tuning, so everyone assumes the safety properties will remain intact afterward. They don’t necessarily.

Fine-tuning changes the model’s behaviour. That’s the entire point of fine-tuning. But when we change behaviour in one area, we can sometimes change behaviour in another area unintentionally. A dataset designed to improve domain expertise can contain examples that weaken refusal patterns, encourage overly confident answers, reproduce harmful stereotypes, or teach the model to follow instructions that conflict with its original safety alignment.

The important point is that this can happen without anyone deliberately trying to bypass the model’s safeguards. Poorly selected training examples can be enough. If the fine-tuning data repeatedly rewards a behaviour that conflicts with the model’s previous safety behaviour, the optimisation process can shift the model toward that behaviour.

So I treat the base model’s safety evaluation as a baseline. Before the fine-tuning run, I record the results of the agreed safety benchmark battery. After fine-tuning, I run the same battery against the fine-tuned model under comparable conditions. Then I compare the results category by category.

What I Compare

I don’t look at one overall safety percentage and call the assessment finished. I want to know where behaviour changed.

• Refusal reliability: Does the model continue to refuse requests that it previously identified as unsafe?
• Policy consistency: Does the model apply the same safety rules across different phrasings and contexts?
• Instruction following: Has fine-tuning made the model more likely to follow conflicting or unsafe instructions?
• Harmful-content resistance: Has performance changed on the organisation’s approved harmful-content test categories?
• Privacy behaviour: Does the model now reveal or reproduce information that the previous model handled more cautiously?
• Bias and fairness: Has the model developed a stronger preference or negative association toward particular groups, products, or categories?
• Over-refusal: Has the model become unnecessarily restrictive in legitimate use cases?
• Consistency across domains: Does improved performance in the target domain come at the expense of safety behaviour elsewhere?

That last point is particularly important. Fine-tuning is usually performed to improve a specific capability. I therefore want to know whether the improvement is isolated to that capability or whether it has caused unexpected behavioural changes outside the intended scope.

The Baseline Comparison

Suppose the base model scores 99% on an agreed safety evaluation and the fine-tuned version scores 95%. I wouldn’t describe the second model as simply “95% safe.” The more useful finding is that the fine-tuning run introduced a measurable regression.

I would then break that four-point difference down. Did the regression come from one category? Did several categories move slightly? Did the model fail completely on a small but important subset of tests? A four-point aggregate change can represent very different security situations depending on where those failures occurred.

For example, a small regression concentrated in a high-impact safety category may deserve more attention than a larger change spread across low-risk edge cases. The numbers tell me that something changed; the category-level results tell me what changed.

Keep the Test Conditions Consistent

The comparison is only useful if I can make a reasonable comparison between the two models. I therefore keep the important evaluation conditions consistent: the same test cases where appropriate, the same evaluation criteria, the same model interaction format, and documented inference settings.

I also preserve the evaluation results rather than simply recording a final score. When a model eventually reaches production, I want to be able to answer a very specific question: What safety behaviour did this model have immediately before deployment, and how did it compare with the previous version?

Test More Than the Original Benchmark

There is another trap here: testing only the benchmark that was used before fine-tuning. That gives me a useful regression signal, but it doesn’t guarantee that the benchmark covers the new risks introduced by the training data.

If the model was fine-tuned on customer-support conversations, for example, I would add evaluation cases that reflect the behaviours introduced by that dataset. If it was trained on internal product documentation, I would test whether it has developed unexpected preferences, unsupported claims, or disclosure behaviour related to that content.

In other words, I use two layers of testing: regression tests to make sure existing safety properties haven’t deteriorated, and domain-specific tests to look for new problems created by the fine-tuning process.

Investigating a Regression

When I find a safety regression, I don’t immediately conclude that the dataset is poisoned. A regression can have several causes: training-data composition, incorrect labels, duplicated examples, changes in training configuration, preprocessing errors, differences in the evaluation environment, or interactions between the fine-tuning method and the original model behaviour.

The next step is therefore investigation. I compare the affected evaluation cases with the training data and dataset version, review what changed between the base and fine-tuned configurations, and determine whether the regression is reproducible.

If removing or correcting a particular subset of training examples causes the behaviour to disappear in a controlled retraining experiment, that gives me much stronger evidence about the source of the regression than simply observing that the model failed a few tests.

Don’t Hide Behind the Average Score

The biggest lesson I want you to take from this is simple: average safety scores can hide important regressions.

A model can improve dramatically on its target task while becoming less reliable in a safety-critical category. That’s why I don’t ask only, “Did fine-tuning improve the model?” I ask two separate questions: What capabilities improved? and What safety properties changed?

Safety degradation testing is therefore best treated as regression detection, not as a one-time pass/fail certification. The objective isn’t to prove that a model is universally “safe.” The objective is to establish whether fine-tuning changed the model’s safety behaviour, identify the categories affected, measure the size of the change, and decide whether that change is acceptable before the model reaches production.

Training Backdoor Probing

A backdoored model can look completely normal during ordinary testing. It can perform well on the intended task, pass standard safety evaluations, and produce sensible answers for normal users. The suspicious behaviour may only appear when a particular condition is present.

That is what makes backdoor testing different from ordinary safety testing. I’m not only asking, “Does the model give an unsafe answer?” I’m asking, “Does the model behave differently when a specific, unusual condition is introduced?”

Research on instruction-tuned language models has demonstrated that training-data poisoning can create conditional behaviours that remain hidden during normal evaluation. The trigger can be associated with a particular phrase, pattern, instruction structure, or other feature of the input.

For an authorised assessment, I therefore treat backdoor probing as a controlled behavioural comparison. I start with a set of ordinary prompts and establish the model’s expected behaviour. I then introduce carefully designed variations and look for statistically or qualitatively unusual changes in the response.

Start With a Clean Baseline

Before looking for a backdoor, I establish what normal behaviour looks like. Ideally, I have access to the original base model as well as the fine-tuned model. I run comparable test cases against both and record the outputs.

This gives me an important reference point. If the fine-tuned model behaves differently from the base model, I can investigate whether the difference is an intended capability improvement or a suspicious conditional behaviour introduced during training.

I also keep the evaluation conditions consistent. Changes in temperature, system instructions, model wrappers, retrieval context, or other inference components can otherwise create behavioural differences that have nothing to do with a training backdoor.

What I Probe For

I look for several categories of anomalous behaviour:

• Trigger sensitivity: Does a small change to an otherwise identical input produce an unexpectedly different response?
• Conditional policy bypass: Does the model follow a different safety policy under a particular input condition?
• Targeted response changes: Does the model consistently produce a particular output pattern when the condition is present?
• Domain-specific behaviour: Does a trigger cause unexpected behaviour only within a particular task or subject area?
• Cross-context persistence: Does the behaviour remain present when the surrounding prompt is changed?
• Robustness to minor variations: Does the suspicious behaviour survive harmless changes in wording or formatting?

I don’t treat one unusual response as evidence of a backdoor. Large language models are probabilistic systems, and isolated anomalies happen. What I’m looking for is repeatable conditional behaviour.

Compare Matched Inputs

One of the most useful techniques is matched-pair testing. I take the same legitimate test case and create controlled variations where only one feature changes.

For example, instead of asking completely different questions, I might compare two semantically equivalent requests where a single authorised test condition has been changed. If the model consistently changes its behaviour only when that condition is present, I have a stronger signal that something unusual is happening.

The important word here is controlled. Randomly generating thousands of prompts makes it difficult to determine why the model behaved differently. Controlled variations allow me to isolate the feature associated with the behavioural change.

Test the Fine-Tuned Model Against the Base Model

When the base model is available, the comparison becomes much more useful. If both models behave normally on ordinary inputs but the fine-tuned model shows a repeatable conditional response that the base model does not, I have a concrete regression to investigate.

I then trace that behaviour back through the fine-tuning lifecycle: dataset version, data sources, preprocessing, training configuration, checkpoints, and evaluation results.

This is especially important for parameter-efficient fine-tuning. Research has found that certain PEFT approaches can remain vulnerable to weight-poisoning backdoors, meaning that updating only a limited set of parameters does not automatically eliminate the security concern.

Don’t Assume the Trigger Is Obvious

A common mistake is to look only for an obvious keyword or suspicious phrase. Backdoors do not necessarily present themselves as a single visible string. Research has explored triggers involving instruction patterns, combinations of features, and other input characteristics.

For that reason, I treat trigger discovery as a behavioural investigation rather than a simple keyword search. I examine unusual clusters in the training data, repeated structures, anomalous examples, and changes in model behaviour that appeared only after fine-tuning.

Measure the Behavioural Difference

Once I find a suspicious condition, I don’t stop at “the model behaved differently.” I measure how reliably the behaviour occurs.

Useful measurements include the proportion of test cases producing the anomalous behaviour, the false-positive rate on similar clean inputs, the consistency of the behaviour across multiple runs, and whether the behaviour transfers across different prompts or contexts.

This helps separate a genuine conditional behaviour from normal model variability. A behaviour that appears once in a hundred trials means something very different from one that appears consistently whenever the same authorised test condition is present.

Investigate the Training Data

If I identify a reproducible anomaly, the next question is where it came from. I compare the affected behaviour against the training dataset and look for examples that could plausibly have taught the model that association.

This is where the dataset audit from earlier in the assessment becomes valuable. I want to connect the behavioural finding to a specific dataset version or source wherever possible. A suspicious training example is much more meaningful when I can establish that it entered the pipeline before the affected model was trained.

Recent research also shows why this deserves serious attention: controlled studies have demonstrated that relatively small numbers of poisoned examples can be sufficient to install backdoors in some fine-tuned classifiers, while the resulting loss of ordinary robustness may be small enough to escape standard evaluation.

Backdoor Probing Is Not a Single Test

I don’t consider a model “backdoor-free” simply because it passes one trigger test. Backdoor behaviour can vary significantly with the model architecture, fine-tuning method, dataset, and attack mechanism. Research surveys distinguish between data-poisoning, weight-poisoning, and other backdoor approaches, which is a useful reminder that no single probing technique covers the entire attack surface.

A stronger assessment therefore combines several signals: dataset provenance, training-run integrity, base-versus-fine-tuned behavioural comparison, controlled probing, regression testing, and investigation of anomalous outputs.

The objective isn’t to prove a mathematical negative — that no backdoor exists anywhere in the model. The practical objective is to increase confidence that the fine-tuning process did not introduce a hidden conditional behaviour that can materially change the model’s security properties.

What a Good Finding Looks Like

If I find something suspicious, I document it as a reproducible security finding rather than simply reporting that “the model may be backdoored.”

The evidence should describe the affected model version, the test condition, the baseline behaviour, the changed behaviour, reproduction rate, affected safety or business property, and any evidence connecting the behaviour to the training pipeline.

That gives the development and security teams something actionable: they can identify the affected dataset or checkpoint, reproduce the issue, determine whether it was intentional or accidental, remove or correct the underlying cause, retrain if necessary, and repeat the evaluation before deployment.

Fine-Tuning Pipeline Access Control

When I assess a fine-tuning pipeline, I don’t stop at the question, “Who has access to the training server?” That’s too narrow. I want to know who can influence the model at every stage of the pipeline.

Fine-tuning is effectively a privileged software-development process for model behaviour. A person who can modify the training dataset, change the training configuration, replace a checkpoint, alter the evaluation process, or approve a model for production may be able to influence what the final model learns. NIST treats the security and resilience of AI systems as including the confidentiality, integrity, and availability of training data, model components, and supporting infrastructure.

That makes the fine-tuning pipeline an access-control problem as much as a machine-learning problem. I therefore map permissions across the complete chain: data source → ingestion → preprocessing → training → model artefact → evaluation → approval → deployment.

Map Who Can Influence Each Stage

I start by identifying the identities, service accounts, automation jobs, and administrators involved in each stage. Then I ask what each one can actually change.

• Data contributors: Who can add or modify content that eventually becomes training data?
• Data engineers: Who can change ingestion, filtering, cleaning, or transformation jobs?
• ML engineers: Who can modify training parameters, code, checkpoints, or model configurations?
• Evaluation owners: Who controls the safety and quality tests used to approve a model?
• Pipeline administrators: Who can modify the CI/CD or orchestration infrastructure?
• Model approvers: Who can promote a trained model from evaluation into production?
• Cloud administrators: Who can access the storage, compute, secrets, and registries supporting the pipeline?

The important thing is to look at the effective permission, not just the role name. Someone called a “data analyst” may appear low risk, but if their account can modify a repository automatically consumed by training, they may have indirect influence over the model.

Separate Dataset Write Access From Training Authority

One control I consider particularly important is separating the ability to modify training data from the ability to start or approve a training run.

If the same person can change the dataset, initiate training, and approve the resulting model, there is very little independent control over the process. A malicious or compromised account could potentially introduce a change and move the resulting model through the pipeline without another person having an opportunity to review it.

A stronger design introduces separation of duties. Data changes can be reviewed and versioned independently, training jobs can consume an immutable dataset version, and production promotion can require an independent approval.

This is essentially applying familiar software-supply-chain principles to model development. NIST’s secure-development guidance for generative AI explicitly extends secure software-development practices into AI model development throughout the software development lifecycle.

Protect the Training Configuration

The dataset isn’t the only thing I protect. Training configuration can also influence the resulting model.

During an assessment, I look at who can modify the training code, hyperparameters, data-selection logic, preprocessing rules, evaluation configuration, model initialization, and checkpoint selection. A pipeline can have excellent dataset controls and still be vulnerable if an unauthorised user can quietly modify the code that determines which examples reach the training job.

I also check whether configuration changes are version-controlled and reviewed. If someone changes a training parameter directly in a production notebook or cloud console and there is no reliable audit trail, reproducing the resulting model becomes much harder.

Service Accounts Matter Too

Human access is only half of the problem. Automated pipeline identities often have much broader permissions than individual users because they need to move data between storage, compute, model registries, and deployment systems.

I therefore inventory service accounts and ask what each one can read, write, execute, or delete. A training job should not automatically receive administrative access to every dataset, model repository, and production environment simply because that makes the pipeline easier to operate.

Where possible, I want narrowly scoped identities for individual stages. The ingestion process should have the permissions it needs to ingest data. The training job should have access to the specific dataset version and required compute resources. The deployment process should be able to retrieve an approved model without being able to rewrite the training history.

Protect Model Weights and Checkpoints

Model checkpoints are another important access-control boundary. If an attacker can replace, delete, or modify a checkpoint after training but before deployment, the organisation may end up deploying a model that was never actually evaluated.

I therefore want model artefacts to be versioned and integrity-protected. The exact model checksum, training configuration, dataset version, and evaluation result should be associated with the model version being promoted.

This creates a useful chain of evidence:

Dataset version → Training run → Model checkpoint → Evaluation result → Approved model → Deployment

If one of those relationships is missing, I ask how the organisation knows that the model evaluated by security is the same model that eventually reached production.

Don’t Let Evaluation Become a Bypass

The evaluation stage deserves its own access controls. Imagine an engineer can modify the model and also modify the tests used to decide whether the model is safe. Even if every change is logged, that creates an obvious conflict of interest.

I therefore check who can add, remove, disable, or alter evaluation cases and who can override a failed result. Evaluation results should be retained as evidence rather than being silently replaced by the results of a later run.

This matters because fine-tuning can introduce training-time security risks that are not visible during ordinary application testing. NIST’s adversarial-machine-learning taxonomy specifically identifies training-stage attacks, including attacks against fine-tuning, as part of the GenAI lifecycle.

Look for Privilege Escalation Paths

During an authorised assessment, I also map indirect privilege paths. A user may not have direct access to the model registry but may be able to modify a pipeline configuration that runs with the registry’s service-account permissions. Another user may not be allowed to change the dataset directly but may be able to modify the ingestion job that produces it.

These indirect paths are often more interesting than the obvious administrator accounts because they can turn a relatively low-privileged identity into an effective model-development identity.

My question becomes: “If this account is compromised, what part of the model lifecycle can it influence?”

Access-Control Evidence I Want

A mature assessment should produce evidence rather than simply confirming that RBAC exists. I want to review role definitions, group memberships, service-account permissions, repository permissions, dataset ACLs, model-registry permissions, pipeline configurations, approval workflows, audit logs, and recent access changes.

I also want to verify that privileged access is actually being used as intended. A permission that exists “just in case” can become a significant risk if nobody reviews it for months.

A Practical Control Model

For a production fine-tuning pipeline, I generally want to see four basic properties:

• Least privilege: Each identity has only the permissions required for its stage of the pipeline.
• Separation of duties: No single identity can silently modify data, train a model, approve the evaluation, and deploy it.
• Strong provenance: Dataset, training run, checkpoint, evaluation, and deployment records are linked and traceable.
• Auditable changes: Important modifications are logged, attributable, reviewable, and difficult to erase or rewrite.

These controls are consistent with the broader NIST approach of managing AI risk throughout the lifecycle rather than treating model security as an isolated testing activity. The AI RMF uses the functions Govern, Map, Measure, and Manage to structure this risk-management process.

The main lesson is simple: protecting the fine-tuning pipeline means protecting the ability to influence model behaviour. If someone can change the data, code, configuration, weights, evaluation, or approval process without appropriate controls, they may have more influence over the production model than their job title suggests.

RLHF and Preference Data Integrity

When I audit a fine-tuning pipeline, I don’t stop at the original training dataset. If the model goes through RLHF or another preference-optimisation stage, I treat the preference data as another security-critical dataset.

The reason is simple: preference data tells the training process which behaviour should be rewarded and which behaviour should be discouraged. If those preferences are inaccurate, systematically biased, manipulated, or deliberately poisoned, the model can learn the wrong objective even when the underlying training infrastructure is working exactly as designed.

This is different from simply asking whether an annotator made a mistake. I want to know whether someone could influence the preference signal at scale and whether the organisation would be able to detect that influence before it affected the model.

Understand the Preference Pipeline

I first map how preference data is created. In a typical workflow, a prompt is given to the model, one or more candidate responses are generated, and a human or automated evaluator indicates which response is preferred. Those preference pairs may then be used to train a reward model or directly optimise the policy, depending on the alignment method.

That creates several security boundaries:

• Prompt generation: Where do the evaluation prompts come from, and who can modify them?
• Candidate generation: Which model version produced the responses being evaluated?
• Annotation: Who or what decides which response is preferred?
• Annotation platform: Who can access, modify, export, or delete preference records?
• Preference processing: What filtering, deduplication, or transformation occurs before training?
• Reward modelling: Which dataset version is used to train the reward model?
• Policy optimisation: Which preference or reward signal ultimately influences the model?

The objective is to establish a complete chain of provenance from prompt → candidate responses → preference decision → processed preference dataset → reward or optimisation stage → resulting model.

Preference Poisoning

Preference poisoning occurs when an attacker or compromised data source manipulates the preference signal so that the training process learns a behaviour the organisation did not intend.

This doesn’t necessarily require changing the underlying model code. If an attacker can influence enough preference examples, they may be able to influence what the reward model considers desirable or what a preference-optimisation method reinforces.

Research has demonstrated this risk experimentally. The Best-of-Venom study found that poisoning a relatively small fraction of preference data could manipulate RLHF models toward targeted behaviours. The researchers specifically investigated poisoning preference pairs used in RLHF and reported successful manipulation with a small percentage of poisoned data in their experimental setting.

More recent research has also examined poisoning against preference-optimisation approaches such as DPO, showing that the problem is not limited to one particular RLHF implementation.

For a security assessment, I therefore ask a practical question: Who has the ability to influence preference data, directly or indirectly, and how much influence could a single compromised source have?

Check for Systematic Preference Bias

Not every integrity problem is malicious. Preference data can become systematically biased because of the annotator population, unclear instructions, organisational incentives, cultural assumptions, or the way the annotation task was designed.

That matters because the model may faithfully learn those preferences.

For example, suppose annotators consistently prefer responses that recommend a particular product, communication style, political framing, or commercial outcome. If the preference dataset contains enough of those decisions, the optimisation process may reinforce the pattern even though nobody explicitly instructed the model to develop that preference.

This is why I examine preference distributions, not just individual annotations. If one label, annotator group, source, topic, or response characteristic dominates the dataset, I want to understand why.

Annotator Integrity Matters

Human annotators are part of the security boundary. I look at how annotators are authenticated, how assignments are distributed, whether annotation activity is logged, and whether unusually high or low agreement rates are investigated.

I also want to know whether the organisation can identify which annotator or annotation batch produced a particular preference record. Without that provenance, investigating a suspicious cluster becomes much harder.

The goal isn’t to assume that annotators are malicious. It is to make the system resilient if an account is compromised, an annotation vendor behaves improperly, or a particular annotation process starts producing systematically unreliable results.

Look for Suspicious Preference Clusters

During an authorised assessment, I look for unusual concentrations in the preference data. Useful investigation signals include sudden changes in preference distributions, repeated identical or near-identical examples, unusual agreement rates, anomalous annotation timing, unexpected correlations with a particular source, and preference patterns that appear only after a particular dataset update.

None of these signals proves poisoning. They are indicators that tell me where to investigate further.

I also compare preference-data versions. If the model’s behaviour changes significantly after a particular preference-data update, I want to be able to identify exactly what changed rather than treating the entire RLHF dataset as one undifferentiated source.

Protect the Reward Model

In traditional RLHF, preference data is commonly used to train a reward model. That creates another important security boundary.

The reward model effectively becomes a judge for model behaviour. If its training data is compromised, the resulting reward signal can encourage behaviour that the organisation never intended to reward.

I therefore treat the reward model as a production-critical model artefact. Its dataset version, training configuration, checkpoint, evaluation results, and integrity information should be tracked just as carefully as the final language model.

This is also why reward hacking deserves attention. NIST describes reward hacking as a situation where an RL-trained system exploits loopholes in the reward definition to obtain high reward through unintended solutions.

A high reward score therefore doesn’t automatically mean that the model learned the intended behaviour. I need to validate whether the rewarded behaviour actually corresponds to the security and quality objectives defined by the organisation.

Separate Preference Integrity From Model Performance

One of the mistakes I avoid is assuming that a preference-trained model is secure simply because its benchmark performance improved.

Performance and integrity answer different questions.

Performance asks: Did the model become better at the target task?

Integrity asks: Did the model improve for the reasons and according to the preferences we intended?

A poisoned preference dataset can potentially produce a model that looks better on selected metrics while simultaneously developing an unwanted behavioural tendency. That is why preference-data security needs independent evaluation rather than relying entirely on the reward score or the same metrics used during optimisation.

Use Provenance and Versioning

For every production-bound preference dataset, I want to be able to answer four questions:

1. Where did these preference records come from?
2. Who or what generated each preference decision?
3. What changed between this dataset version and the previous version?
4. Which model or reward-model training run consumed this exact version?

NIST guidance on AI security emphasises provenance and integrity controls for datasets and models as part of addressing poisoning and other adversarial-ML risks.

In practice, that means retaining dataset versions, manifests, hashes or equivalent integrity records, annotation metadata, training-run identifiers, and evaluation results. The exact implementation will vary, but the principle is the same: don’t allow the preference signal to become an untraceable black box.

Test the Resulting Behaviour

Even with strong preference-data controls, I still test the resulting model. Data-integrity controls reduce the likelihood of poisoning, but they don’t prove that the final model learned exactly what was intended.

I compare the base model and preference-trained model across the important behavioural categories. I look for unexpected changes in refusal behaviour, bias, instruction following, factual reliability, commercial preferences, privacy behaviour, and other properties relevant to the model’s intended use.

This creates a second line of defence. The first line protects the preference data. The second checks whether anything undesirable actually appeared in the model.

What I Want From an RLHF Integrity Assessment

At the end of the assessment, I want more than a statement that “RLHF was reviewed.” I want evidence that the preference pipeline has appropriate controls around provenance, access, annotation integrity, dataset versioning, reward-model integrity, and behavioural validation.

The complete security chain should look something like this:

Preference source → Annotation → Preference dataset → Reward/optimisation process → Model checkpoint → Safety evaluation → Production approval

If I cannot trace that chain, I have a visibility problem. If someone can modify several stages without independent review, I have an access-control problem. And if a small change in preference data can produce a large, unexplained behavioural shift, I have a model-integrity problem that deserves investigation.

The key lesson is that RLHF doesn’t remove the training-data security problem — it creates another layer of it. The model is learning not only from the original fine-tuning examples but also from the preferences used to tell it which behaviours are desirable. Protecting that preference signal is therefore part of protecting the model itself.

Post-Fine-Tuning Security Checklist

I treat the post-fine-tuning security review as the production gate for the new model version. Fine-tuning has changed the model, so I don’t assume that the security properties of the previous version automatically carry over.

Think of it the same way you would think about a security review before deploying a major software release. The model may have better accuracy, better domain knowledge, and better task performance, but those improvements don’t tell me whether the training process introduced a new security problem.

NIST’s AI RMF recommends that AI systems be tested before deployment and regularly while operating, with documented metrics, independent assessment where appropriate, and evaluation of security, safety, privacy, fairness, and other relevant trustworthiness characteristics.

For that reason, I run this checklist after every production-bound fine-tuning cycle. The exact duration depends on the size of the model and the existing test suite. The important part isn’t whether the review takes one hour or one day; it’s that the model cannot bypass the gate simply because the training run completed successfully.

1. Confirm Exactly What Was Trained

Before I test the model, I establish exactly what I’m testing. I record the model version, base-model version, fine-tuning method, dataset version, training-run identifier, training configuration, checkpoint identifier, and relevant code or pipeline version.

This sounds administrative, but it is a security control. If the model later behaves unexpectedly, I need to reconstruct exactly which data and configuration produced it.

• Base model: Which exact model and version was used?
• Dataset: Which immutable dataset version was consumed?
• Training run: Which job produced the checkpoint?
• Configuration: Which important training parameters were used?
• Code: Which preprocessing and training code version was executed?
• Checkpoint: Which exact model artefact is being evaluated?

If I cannot establish this chain, I consider that a provenance problem before I even start the behavioural tests.

2. Verify Dataset Integrity

Next, I verify that the dataset used for training is the dataset that was approved for training.

I compare the dataset manifest or equivalent integrity record, verify the expected version, review significant changes from the previous version, and check whether the documented data sources match what actually entered the training pipeline.

NIST’s GenAI secure-development guidance specifically recommends verifying the provenance and integrity of training, testing, fine-tuning, and alignment data and analysing data for indicators such as poisoning, bias, homogeneity, and tampering.

I don’t need to manually inspect every training example. The goal is to establish that the dataset has appropriate provenance and that there are no unexplained changes that should prevent the model from reaching production.

3. Run Safety Regression Tests

This is where I compare the fine-tuned model against the base model using the established safety test suite.

I record both the absolute results and the change between versions. A model that passes a safety test today isn’t necessarily acceptable if fine-tuning caused a significant regression from the previous version.

I look at categories individually rather than relying only on one aggregate score. That lets me identify whether the regression is concentrated in refusal behaviour, privacy, bias, harmful-content handling, instruction following, or another relevant category.

4. Run Backdoor and Conditional-Behaviour Tests

The next question is whether fine-tuning introduced behaviour that appears only under particular conditions.

I use controlled behavioural probes and compare matched inputs to identify unexpected conditional changes. A single unusual response isn’t enough to call something a backdoor. I’m looking for behaviour that is reproducible and materially different from the base model.

If a suspicious pattern appears, I preserve the exact inputs, outputs, model version, inference configuration, and reproduction rate so the result can be investigated rather than dismissed as an isolated anomaly.

5. Test for Dataset-Induced Bias

Fine-tuning can reproduce patterns present in the training data, including patterns that were never intended to become model behaviour.

I therefore test the model for domain-specific biases that weren’t present, or weren’t as strong, in the base model. Depending on the application, this could include demographic bias, commercial preference, geographic bias, language bias, or systematic preference for particular outcomes.

This is particularly important when the fine-tuning dataset contains internal documents, customer interactions, sales material, or human-generated examples. A model can reproduce an organisational assumption without anyone explicitly instructing it to do so.

6. Validate the Intended Capability Improvement

Security regression testing shouldn’t become an excuse to ignore the reason the model was fine-tuned in the first place.

I verify that the model actually improved on its intended task. I compare it against the base-model baseline using the agreed domain metrics and representative test cases.

The important comparison is therefore two-dimensional:

Did the intended capability improve?

Did security or trustworthiness degrade?

A model that becomes substantially more capable but introduces an unacceptable security regression should not automatically pass. Likewise, a model that is extremely safe but fails the business requirement isn’t a successful fine-tuning result.

7. Check Access-Control and Pipeline Evidence

I also review the pipeline evidence around the training run. Who could modify the dataset? Who could change the training configuration? Who could start the job? Who could replace the checkpoint? Who could modify the evaluation? Who approved production deployment?

This gives me confidence that the model wasn’t only technically tested but also produced through a controlled process.

NIST’s secure-development guidance applies secure software-development practices throughout the AI model lifecycle, including controls around development environments, provenance, integrity, testing, and release processes.

8. Verify the Evaluated Model Is the Deployed Model

This is one of the easiest checks to overlook.

I want a reliable relationship between the model I evaluated and the model that will actually be deployed. The model identifier, checkpoint or digest, configuration, and evaluation record should all correspond to the same artefact.

Otherwise, an organisation can end up with a perfectly clean evaluation report for Model A while Model B is what actually reaches production.

The release chain should therefore look something like:

Approved dataset → Training run → Model checkpoint → Security evaluation → Approval → Deployment

9. Review the Failure Budget

I don’t recommend treating every failed test as an automatic production blocker. Some failures may be known limitations, false positives, or low-risk edge cases.

Instead, I define the acceptable failure threshold before reviewing the final results. More importantly, I identify categories where any unexplained regression is unacceptable.

For example, an organisation may tolerate a small change in a low-impact benchmark while treating a reproducible privacy regression or newly introduced security bypass as a release blocker.

The decision should therefore be based on documented risk tolerance rather than whoever happens to be reviewing the model that day.

10. Document Exceptions

If the model passes with known limitations, I document those limitations explicitly.

The record should explain what failed, why the failure is currently considered acceptable, what compensating controls exist, who accepted the residual risk, and whether the issue requires follow-up after deployment.

This prevents a temporary exception from quietly becoming a permanent vulnerability.

11. Define the Rollback Point

Before deployment, I want to know what happens if the model causes problems in production.

The previous model version should remain identifiable and deployable, and the organisation should know which configuration and data version produced it. That gives the team a practical rollback path if post-deployment monitoring identifies a security regression.

NIST’s AI RMF also emphasises post-deployment monitoring, incident response, recovery, change management, and continual improvement as part of managing AI risk.

12. Record the Final Security Decision

At the end of the review, I want an explicit decision rather than a collection of test results.

• APPROVED: No unacceptable regressions identified and required evidence is complete.
• APPROVED WITH CONDITIONS: Known limitations exist, but documented controls and risk acceptance are in place.
• BLOCKED: A significant security regression, integrity issue, unexplained behavioural change, or missing control requires remediation.

That decision should be associated with the exact model version and retained as part of the model’s release record.

The Practical Checklist

• ☐ Base model and fine-tuned model versions recorded
• ☐ Exact training dataset version identified
• ☐ Dataset provenance and integrity verified
• ☐ Training configuration and code version recorded
• ☐ Training-run and checkpoint identifiers recorded
• ☐ Safety regression testing completed
• ☐ Domain capability testing completed
• ☐ Bias and unexpected-behaviour testing completed
• ☐ Backdoor/conditional-behaviour probing completed where appropriate
• ☐ Privacy and data-leakage testing completed where relevant
• ☐ Fine-tuning pipeline access reviewed
• ☐ Evaluation results independently reviewed where required
• ☐ Evaluated model verified against deployment artefact
• ☐ Known exceptions documented and risk accepted
• ☐ Rollback model identified
• ☐ Final production decision recorded

Why This Gate Matters

The biggest mistake is to treat a successful training run as a successful security outcome. Training completion only tells me that the optimisation process finished. It doesn’t tell me whether the model learned the right things, whether the dataset was trustworthy, whether safety properties were preserved, or whether the final checkpoint is the one that was actually evaluated.

I therefore think of every fine-tuning run as a new model release, not as a minor configuration change. The model needs its own provenance, evaluation evidence, security decision, and rollback path.

That mindset turns fine-tuning security from a one-time assessment into a repeatable release-control process. And that’s the real objective of this checklist: don’t let a model change reach production simply because nobody noticed that its security properties changed.