Language Model Security Database

Vision Language Models (VLMs) are vulnerable to visual prompt injection attacks via text-to-image obfuscation. While these models often possess safety guardrails for standard text-based inputs, they fail to apply equivalent safety alignment to textual instructions embedded visually within an image. An attacker can overlay malicious instructions (e.g., requests for illegal acts, hate speech) onto an image file and submit it to the model. The model’s Optical Character Recognition (OCR) or visual…

Large Language Model (LLM) based search engines utilizing Retrieval-Augmented Generation (RAG) are vulnerable to ranking manipulation attacks via indirect prompt injection. Adversaries can embed optimized adversarial triggers or crafted semantic patterns within external webpage content. When these manipulated documents are retrieved and integrated into the LLM's context window alongside a user query, the adversarial content disrupts the model's contextual understanding. This results in the LLM…

Large Language Model (LLM) tool-calling systems are vulnerable to adversarial tool injection attacks. Attackers can inject malicious tools ("Manipulator Tools") into the tool platform, manipulating the LLM's tool selection and execution process. This allows for privacy theft (extracting user queries), denial-of-service (DoS) attacks against legitimate tools, and unscheduled tool-calling (forcing the use of attacker-specified tools regardless of relevance). The attack exploits vulnerabilities…

A vulnerability in LLM-based agents, dubbed AI Agent Injection (AI²), allows attackers to hijack the agent's actions by manipulating the agent's memory retrieval mechanism. The attack involves two main steps: (1) Stealing action-aware knowledge from the agent's memory using crafted adversarial queries targeting the retriever module and (2) Generating Trojan prompts consisting of a Trojan string and hijacking instructions. The Trojan string is designed to manipulate the retriever into…

LLM-based relevance assessment frameworks, such as the Umbrela system, are vulnerable to evaluation subversion and artificial score inflation due to evaluation circularity and LLM "narcissism" (an LLM's inherent bias toward favoring LLM-generated outputs). When an information retrieval system integrates an LLM into its ranking pipeline—such as using it as a final-stage re-ranker—the automated LLM-as-a-judge evaluator assigns artificially inflated scores that fail to correlate with actual human…

Jailbreaking vulnerabilities in Large Language Models (LLMs) used in Retrieval-Augmented Generation (RAG) systems allow escalation of attacks from entity extraction to full document extraction and enable the propagation of self-replicating malicious prompts ("worms") within interconnected RAG applications. Exploitation leverages prompt injection to force the LLM to return retrieved documents or execute malicious actions specified within the prompt.

A vulnerability in Retrieval-Augmented Generation (RAG) systems utilizing LangChain allows for indirect jailbreaks of Large Language Models (LLMs). By poisoning the external knowledge base accessed by the LLM through LangChain, attackers can manipulate the LLM's responses, causing it to generate malicious or inappropriate content. The attack exploits the LLM's reliance on the external knowledge base and bypasses direct prompt-based jailbreak defenses.