AI Output XSS Lab: When the Chatbot's Reply Is Trusted HTML

This article describes a practical security vulnerability where applications render large language model (LLM) responses directly as HTML without sanitization, enabling stored XSS attacks. An attacker can manipulate the model's output to include malicious HTML elements with event handlers that execute JavaScript in the victim's browser. The vulnerability represents a fundamental misunderstanding of trust boundaries—treating model output as safe when it should be considered untrusted user-influenced content.

• LLM output enables classic XSS via trusted sinkWhen applications use innerHTML to render model replies for rich formatting without sanitization, they create a direct XSS vector. Because users can influence what the model outputs through their prompts, an attacker can steer the model toward including malicious markup—such as an image element with a broken source and an error handler—which executes when the reply is rendered into the DOM.
• Model output is untrusted user-influenced contentThe core security principle is that LLM responses are not safe by default. Users shape model outputs through their input, making the model a delivery channel for attacker-controlled payloads rather than a source of trustworthy data. Rendering any user-influenced string as HTML without sanitization—whether from a form field or a model reply—is equally dangerous.
• Attack requires three sequential stepsThe lab demonstrates a structured approach: first confirm the reply is rendered as HTML rather than escaped text; second, craft a prompt that causes the model to output an HTML element with a load-time event handler; third, ensure the handler calls a JavaScript function to demonstrate code execution. This concrete methodology shows how attackers methodically exploit the vulnerability.
• Common in production as rich formatting spreadsThe article identifies this vulnerability as OWASP LLM05 Improper Output Handling and notes it has become a common, high-impact finding in real products. Applications rushing to implement rich formatting from model outputs often skip sanitization steps, making this an actively exploited vulnerability class in deployed systems.
• DOMPurify and markup sanitization as primary defenseThe recommended mitigation is sanitizing model output with a trusted library like DOMPurify before any innerHTML rendering. Alternative approaches include rendering only as plain text or Markdown with HTML disabled, and adding a strict Content-Security-Policy as a backstop—but sanitization is the primary control when HTML rendering is required.
• Event handlers bypass naive escaping approachesSimply trying different HTML elements when basic escaping is present shows that event handlers (like onerror on images) will execute even when standard text escaping is attempted. The vulnerability persists because the application still renders as HTML—it just changes which payload element succeeds.

Bottom line: Treat all LLM output as untrusted user-influenced content and never pass it directly to HTML rendering sinks like innerHTML without sanitization via a trusted library like DOMPurify. Model output is simply another untrusted data source feeding an unsafe sink—the fact that it comes from an AI model changes the delivery mechanism but not the fundamental XSS risk.