Language Model Security Database

Large Language Model (LLM) based search engines utilizing Retrieval-Augmented Generation (RAG) are vulnerable to ranking manipulation attacks via indirect prompt injection. Adversaries can embed optimized adversarial triggers or crafted semantic patterns within external webpage content. When these manipulated documents are retrieved and integrated into the LLM's context window alongside a user query, the adversarial content disrupts the model's contextual understanding. This results in the LLM…

Voting-based Large Language Model (LLM) leaderboards, such as Chatbot Arena, are vulnerable to adversarial ranking manipulation due to insufficient response anonymity. While these systems obscure model identities during head-to-head comparisons to prevent bias, an attacker can de-anonymize the models with high accuracy (>95%) by analyzing response content. The attack functions in two stages: (1) Re-identification, where the attacker submits specific prompts (identity-probing or stylometric…

Large Language Models (LLMs) are vulnerable to multi-turn adversarial attacks that skillfully decompose malicious requests into seemingly benign interactions, progressively guiding the dialogue towards harmful outputs. This vulnerability allows attackers to bypass LLM safety mechanisms through a series of strategically crafted prompts, exploiting the model's iterative response generation. The attack's success hinges on dynamically adapting each prompt based on the LLM's previous responses…

Large Language Models (LLMs) are vulnerable to malicious prompts disguised as summaries of scientific papers, even when those papers are fabricated by the attacker. This allows attackers to manipulate LLMs into generating responses exhibiting significantly increased stereotypical bias and toxicity. The vulnerability is exacerbated by multi-turn interactions, where bias scores tend to increase with each subsequent response. The inclusion of author names and publication venues in the fabricated…

Large Language Models (LLMs) are vulnerable to a self-instruct few-shot jailbreaking attack that leverages pattern and behavior learning to bypass safety mechanisms. The attack efficiently induces harmful outputs by injecting a strategically chosen response prefix into the model's prompt and exploiting the model's tendency to mimic co-occurrence patterns of special tokens preceding the prefix. This allows the attacker to elicit unsafe responses with a small number of carefully crafted…

Large language models (LLMs) are vulnerable to adversarial prompt engineering attacks that leverage proximity constraints to elicit harmful behaviors. By subtly modifying benign prompts within a semantically close embedding space, attackers can bypass existing safety mechanisms and induce undesired outputs, even when the original prompts would not trigger such a response. This vulnerability exploits the model's sensitivity to small perturbations in the input embedding, resulting in the…

A vulnerability in LLM-based agents, dubbed AI Agent Injection (AI²), allows attackers to hijack the agent's actions by manipulating the agent's memory retrieval mechanism. The attack involves two main steps: (1) Stealing action-aware knowledge from the agent's memory using crafted adversarial queries targeting the retriever module and (2) Generating Trojan prompts consisting of a Trojan string and hijacking instructions. The Trojan string is designed to manipulate the retriever into…

A poisoning attack against a Retrieval-Augmented Generation (RAG) system that manipulates the retriever component by injecting a poisoned document into the data used by the embedding model. This poisoned document contains modified and incorrect information. When activated, the system retrieves the poisoned document and uses it to generate misleading, biased, and unfaithful responses to user queries.

Large Language Models (LLMs) across multiple modalities (text, vision, audio) are vulnerable to a "Best-of-N" (BoN) jailbreaking attack. This attack repeatedly submits slightly modified versions of a harmful prompt (e.g., text with altered capitalization, images with modified text style, audio with altered pitch or speed) until a safety mechanism is bypassed and a harmful response is elicited. The effectiveness of the attack scales with the number of attempts (N). While individual…

A bimodal adversarial attack, PBI-Attack, can manipulate Large Vision-Language Models (LVLMs) into generating toxic or harmful content by iteratively optimizing both textual and visual inputs in a black-box setting. The attack leverages a surrogate LVLM to inject malicious features from a harmful corpus into a benign image, then iteratively refines both image and text perturbations to maximize the toxicity of the model’s output as measured by a toxicity detection model (Perspective API or…