AI Output Schema Bypass Lab: Valid JSON, Impossible Values

This article describes a security vulnerability in AI systems where a backend accepts LLM-generated structured output as valid merely because it passes JSON schema validation. The lab demonstrates how an attacker can craft an expense approval that has correct data types and required fields but violates business rules, such as approving a $1,000,000 expense when policy caps it at $5,000. The attack matters because schema validation only confirms structural correctness, not whether values obey actual constraints, creating a gap that teams deploying LLMs for decisions commonly fail to close.

• Schema validation checks shape, not senseJSON schema validation confirms that output contains the right field types and required keys—for example, that 'approved' is a boolean and 'amount' is a number. It does not verify whether those values obey business rules, such as whether the amount respects a policy cap of $5,000. The article shows ApprovalBot emits {approved, amount, reason}, and the backend accepts any schema-valid version without checking the cap.
• Attackers steer models to violate constraintsBecause an attacker can influence the LLM's output through prompt engineering or other steering techniques, they can craft a verdict that is structurally perfect but policy-breaking. The attack approach confirms the backend validates structure but not the policy cap, then deliberately induces ApprovalBot to emit approved: true with amount above 100,000, all within valid JSON types and required fields.
• Valid JSON is not a valid business decisionThe core lesson the lab teaches is that 'valid JSON' is not 'valid decision.' The validator answers only 'is this the right shape?' and stops there. The dangerous part—whether the numbers obey policy—is never checked, allowing an out-of-policy approval to pass validation and be acted upon.
• Vulnerability maps to OWASP LLM05The article identifies this attack as an instance of OWASP LLM05 Improper Output Handling, with an Excessive Agency angle when the decision triggers an action. This is described as a growing mistake as teams use LLMs for decisions and return structured output, trusting schema-valid output as correct.
• Business rules must be enforced in codeDefense requires encoding business constraints directly in the validator—maximum amounts, allowed enums, and cross-field consistency rules—rather than relying on the schema or assuming the model will self-correct. Type-checking the JSON is explicitly not the same as enforcing the policy.
• Treat model decisions as suggestions, not verdictsThe article recommends treating the model's decision as a suggestion that is re-checked by deterministic code before any action is taken. This approach adds a layer of verification between the LLM output and the backend action, reducing the impact of any policy violation the schema misses.
• Human review threshold for high-impact decisionsThe defense strategy includes keeping high-impact approvals under human review above a threshold and logging or alerting on verdicts that approach or exceed policy limits. This creates a final checkpoint for decisions that carry significant risk or cost.

Bottom line: Schema validation is necessary but never sufficient to ensure an LLM's structured output complies with business rules; teams must add value-level validation in code to enforce actual constraints like caps and enums before acting on the model's decision. The lab demonstrates that an attacker can easily produce schema-valid JSON that violates policy, so backend systems must never assume that correct JSON structure implies a correct business decision.