Language Model Security Database

Voting-based Large Language Model (LLM) leaderboards, such as Chatbot Arena, are vulnerable to adversarial ranking manipulation due to insufficient response anonymity. While these systems obscure model identities during head-to-head comparisons to prevent bias, an attacker can de-anonymize the models with high accuracy (>95%) by analyzing response content. The attack functions in two stages: (1) Re-identification, where the attacker submits specific prompts (identity-probing or stylometric…

Large Language Models (LLMs) employing reinforcement learning from human feedback (RLHF) for safety alignment are vulnerable to a novel "alignment-based" jailbreak attack. This attack leverages a best-of-N sampling approach with an adversarial LLM to efficiently generate prompts that bypass safety mechanisms and elicit unsafe responses from the target LLM, without requiring additional training or access to the target LLM's internal parameters. The attack exploits the inherent tension between…

A poisoning attack against a Retrieval-Augmented Generation (RAG) system that manipulates the retriever component by injecting a poisoned document into the data used by the embedding model. This poisoned document contains modified and incorrect information. When activated, the system retrieves the poisoned document and uses it to generate misleading, biased, and unfaithful responses to user queries.

A vulnerability exists in Large Language Model (LLM)-based time series forecasting architectures, specifically affecting models such as TimeGPT, LLMTime, and TimeLLM. These models are susceptible to a gradient-free, black-box adversarial attack method termed Directional Gradient Approximation (DGA). An attacker can inject imperceptible perturbations into the historical time series input window (lookback window) to manipulate the model's output. By treating the model as a black box and…

Large language models (LLMs) are vulnerable to adversarial suffix injection attacks. Maliciously crafted suffixes appended to otherwise benign prompts can cause the LLM to generate harmful or undesired outputs, bypassing built-in safety mechanisms. The attack leverages the model's sensitivity to input perturbations to elicit responses outside its intended safety boundaries.

Large Language Models (LLMs) employing safety alignment mechanisms are vulnerable to a bypass attack using simple, stochastic random augmentations of input prompts. The attack leverages the inherent brittleness of safety alignment to minor, randomly introduced modifications in the input, causing the LLM to generate unsafe outputs despite its safety training. Character-level augmentations prove significantly more effective than string insertions.

Large Language Models (LLMs) are vulnerable to a novel jailbreak attack that exploits resource limitations. By overloading the model with a computationally intensive preliminary task (e.g., a complex character map lookup and decoding), the attacker prevents the activation of the LLM's safety mechanisms, enabling the generation of unsafe outputs from subsequent prompts. The attack's strength is scalable and adjustable by modifying the complexity of the preliminary task.

PathSeeker demonstrates a novel black-box jailbreak attack against Large Language Models (LLMs) that utilizes multi-agent reinforcement learning. The attack iteratively modifies input prompts based on model responses, leveraging a reward mechanism focused on vocabulary expansion in the LLM's output to circumvent safety mechanisms and elicit harmful responses. This technique bypasses existing safety filters by encouraging the model to relax its constraints, rather than directly targeting…

Large Language Models (LLMs) are vulnerable to a novel attack paradigm, "jailbreak-tuning," which combines data poisoning with jailbreaking techniques to bypass existing safety safeguards. This allows malicious actors to fine-tune LLMs to reliably generate harmful outputs, even when trained on mostly benign data. The vulnerability is amplified in larger LLMs, which are more susceptible to learning harmful behaviors from even minimal exposure to poisoned data.

LLM-based autonomous agents are vulnerable to malfunction amplification attacks. These attacks exploit the inherent instability of agents by inducing repetitive or irrelevant actions through various methods including prompt injection and adversarial perturbations, leading to agent malfunction and task failure. The attacks do not rely on overtly harmful actions, making them harder to detect with standard LLM safety mechanisms.