Embedding Poisoning Lab: Outrank the Real Document in RAG

This article presents a hands-on lab demonstrating embedding poisoning, an attack where adversaries craft documents optimized for high similarity scores to outrank legitimate sources in RAG (retrieval-augmented generation) systems. The attack parallels SEO spam but targets AI systems, exploiting the fact that retrievers rank by similarity, not authority, allowing attackers to inject false information into model responses. This matters because RAG systems increasingly power enterprise applications, and the technique is technically simple yet widely undefended against.

• Ranking determines model truth in RAGRAG systems return the top-k documents ranked by similarity to the query and feed them into the model's context. Whatever ranks highest becomes the factual basis for the model's response. An attacker who controls ranking controls what the model believes and outputs, regardless of the actual truth or source authority.
• Similarity scoring is decoupled from authorityThe retriever measures how closely a document matches query terms and semantics, not whether the content is accurate or from a trusted source. This fundamental decoupling means an attacker can write dense, keyword-optimized text that scores high on similarity without any legitimate claim to authority or accuracy.
• Attack technique uses keyword and semantic densityThe practical attack repeats exact query terms more frequently than the legitimate document, adds synonyms (e.g., 'return,' 'reimbursement,' 'money-back' for 'refund'), and wraps the false claim in an authoritative-sounding title like 'Updated Refund Policy 2024.' This combination exploits both keyword overlap and modern embedding models' treatment of semantic similarity.
• Real-world validation through academic researchThe article cites PoisonedRAG (2024) as evidence that a small number of optimized documents can control answers in realistic RAG setups. Real embeddings are harder to game than simple keyword overlap, but the attacker's fundamental approach—optimizing for retriever ranking—remains effective across different embedding models.
• Many RAG pipelines lack spam defensesThe article states that keyword stuffing is crude against modern search engines, but 'many RAG pipelines have no spam defences at all.' This defense gap means the attack succeeds not because the technique is sophisticated, but because systems assume documents in their index are trustworthy or that ranking is inherently reliable.
• Defense requires authentication and re-rankingRecommended defenses include: authenticating sources by indexing only from trusted repositories, re-ranking with a second model that scores both relevance and authority separately, carrying provenance metadata into the model prompt to favor official sources, and flagging anomalous keyword density or near-duplicate 'updated' documents.
• Attack surface expands with small top-k valuesWith a retriever's top-k set to 3 documents, an attacker only needs to outrank two legitimate sources to dominate the answer. Smaller top-k values reduce the attacker's work and increase the likelihood of poisoning success for targeted queries.

Bottom line: Embedding poisoning is a practical, underdefended attack that exploits RAG retrievers' reliance on similarity scoring over source authority; defenders must implement multi-layered controls including source authentication, dual-model re-ranking, and provenance tracking rather than assuming the retrieval ranking is reliable on its own.