LLM02 Sensitive Information Disclosure — How LLMs Leak PII, Credentials & System Data | AI LLM Hacking Course Day 6

In Day 5 you demonstrated how injection payloads reach the AI through documents, web pages, and emails. Today the focus shifts to what those injections reveal. LLM02 overlaps with LLM07 System Prompt Leakage at the system prompt layer, but extends significantly further — into training data, RAG knowledge bases, and cross-user session data. The distinction matters for your report: LLM07 is the mechanism, LLM02 is the impact.

Four LLM02 Disclosure Mechanisms

LLM02 manifests through four distinct mechanisms, each with a different attack path and a different mitigation. Understanding which mechanism produced a finding determines how you write the remediation section and what evidence you need for the report.

System prompt leakage. The developer’s instruction set is extracted via LLM01 injection or LLM07 extraction techniques. The system prompt often contains far more than intended — credentials, architecture details, employee names, internal URLs. This is the most consistently high-severity variant because developers routinely embed operational context in system prompts without considering extractability.

Training data memorisation. The model has encoded specific text from its training corpus in its weights. With the right prompt context — partial sequences that match memorised text — the model reproduces real data: email addresses, phone numbers, code snippets, even cryptographic keys that appeared in training data. This vector targets the model itself, not the application deployment.

RAG retrieval exposure. The RAG knowledge base contains sensitive documents that should not be accessible to the querying user. A carefully constructed query retrieves those documents via the vector search, and the model incorporates their content into its response. This is often an access control problem — the knowledge base was not partitioned by user permission level.

Cross-session leakage. Application-level conversation history storage creates cross-session or cross-user exposure. If conversation history is not properly isolated — if one user’s session data can influence another user’s context — information disclosed in one session can be retrieved in another. This is not a model-level vulnerability but an application architecture vulnerability that the model’s context window makes exploitable.

System Prompt Credential Extraction

Credential disclosure in system prompts is more common than any client expects. On the first five AI red team engagements I ran, four of them had credentials in the system prompt. Not always AWS keys — sometimes internal API tokens, sometimes database passwords, sometimes just internal hostnames that are themselves sensitive — but consistently more than the developer believed they had put there.

The attack is sequential. Extract the system prompt (LLM07 techniques from Day 18). Scan the output for credential patterns. Verify each credential is real by testing it against its target service — but only within scope. Document the credential type, the service it accesses, and the permissions that service grants. The CVSS score reflects the worst-case impact of the most permissive credential, not the average.

Training Data Memorisation — Surfacing PII

Training data memorisation is the LLM02 variant that targets the model itself rather than the application deployment. Research consistently demonstrates that large language models memorise specific sequences from their training data — sometimes verbatim — and can reproduce them when prompted with the right context. For models trained on internet data, this includes real names, email addresses, phone numbers, and occasionally more sensitive information that appeared in the training corpus.

The test approach is probabilistic rather than deterministic. You are not looking for a single reliable exploit but probing for the presence of memorised content by constructing prompts that might trigger memorised completions. The most effective technique is prefix completion: provide the beginning of a sequence that appeared in training data and observe whether the model completes it with real information. For public figures and organisations with extensive online presence, their contact details and internal documents that were public at training time are the most likely memorised candidates.

RAG and Context Window Data Exposure

RAG-based LLM02 disclosure occurs when the knowledge base contains documents that a given user should not be able to access, but the vector search retrieves them in response to a crafted query. The root cause is almost always an access control gap — the knowledge base was built without user-level partitioning, treating all embedded documents as equally accessible to all users of the system.

The test approach mirrors a vertical privilege escalation test. Identify what type of documents exist in the knowledge base. Construct queries designed to retrieve documents above your permission level — confidential policy documents, other users’ data, financial records, executive communications. Observe whether the RAG retrieval enforces access controls or returns whatever the vector search identifies as most semantically similar.

Cross-Session and Cross-User Leakage

LLMs are stateless between API calls — they have no memory of previous conversations at the model level. But applications that store conversation history and include it in subsequent context windows create a state layer that can produce cross-session leakage. If conversation history from session A is accessible or influenceable by session B, information disclosed in session A can be extracted in session B.

I have found two implementation patterns that produce this vulnerability. First: shared system prompts that include user-specific data for all sessions. When a developer puts user-specific context (“the current user is John Smith, employee ID 4421”) in the system prompt and that system prompt is shared or cached across sessions, one session’s user data bleeds into another session’s context. Second: conversation summaries stored in shared knowledge bases. When an AI summarises conversations and stores those summaries in the RAG knowledge base without user-level partitioning, another user can retrieve those summaries through crafted queries.

Calculating Impact — Regulatory and Business Severity

LLM02 severity is not set by the CVSS base score alone — regulatory exposure significantly changes how clients respond to the finding. A database connection string that provides access to customer records is a Critical CVSS finding, but it is also potentially a GDPR Article 33 breach notification event, a HIPAA reportable incident, or a PCI-DSS violation. Those regulatory frameworks impose timelines and requirements that dwarf the technical remediation.

My practice for every LLM02 finding: calculate the CVSS base score, identify the highest applicable regulatory framework, and write the business impact in terms of both the technical capability (what an attacker can access) and the regulatory obligation (what the client must do within 72 hours under GDPR if this credential was used). That dual framing is what moves an LLM02 finding from a developer task to a board-level notification.