AI Training Data Extraction — Memorised Secrets
TicketGPT was fine-tuned on five years of raw support tickets, and the scrubbing was 'best effort'. It won't hand out secrets when asked. But models memorise rare strings from fine-tuning data — give it the right prefix and it may autocomplete a ticket it saw in training, secrets included.
What is training data extraction?
Models memorise some of their training data word for word, especially text that is unusual or appeared many times. A model fine-tuned on raw support tickets may have memorised the passwords and card details customers pasted into them.
Training data extraction is getting the model to reproduce that text. A direct request is refused. Asking it to continue a familiar passage or complete a record often is not, because that is the task it was trained on.
A real case
We have not found a public breach from 2025 or 2026 attributed to training data extraction, so we are not listing one. Researchers have shown repeatedly that models can reproduce memorised training text, which is why OWASP covers it under LLM02.
How to fix it
- Remove secrets and personal data from training sets with proper tooling before training.
- Avoid fine-tuning on raw logs, tickets or chat transcripts.
- Scan outputs for patterns such as keys, card numbers and emails.
- Use privacy-preserving training methods for sensitive datasets.
Questions people ask
Do models really memorise their training data?
Yes, in part. Most output is newly generated, but rare or repeated strings can be reproduced exactly.
Is fine-tuning riskier than a base model?
Often, because fine-tuning sets are small and specific, which makes memorisation more likely.
Can memorised data be removed afterwards?
Not reliably. The dependable fix is to retrain on cleaned data.
Read the full OWASP LLM Top 10 guide · All free labs · AI red team certification