Language Model Security Database

The GAP framework, as described in arXiv:2501.18638, reveals vulnerabilities in various large language models (LLMs) by generating stealthy jailbreak prompts that bypass content moderation systems. The framework leverages a graph-based attack strategy, enabling knowledge sharing across attack paths for enhanced efficiency and evasion. This allows the successful bypassing of multiple LLM safety mechanisms, including those based on perplexity and prompt-based heuristics.

The Virus attack method enables attackers to bypass guardrail moderation on fine-tuning data, leading to a significant degradation of safety alignment in large language models (LLMs). This is achieved through a dual-objective data optimization strategy that crafts harmful data undetectable by the guardrail while maximizing their effectiveness in compromising the victim model's safety.

Large language models (LLMs) exhibit increased responsiveness to prompts framed within positive narratives. The Happy Ending Attack (HEA) exploits this by embedding malicious requests within a positive-sentiment scenario culminating in a happy ending. This allows the LLM to generate responses that fulfill the malicious request while perceiving the overall prompt as benign.

Large Language Models (LLMs) used in hate speech detection systems are vulnerable to adversarial attacks and model stealing, resulting in evasion of hate speech detection. Adversarial attacks modify hate speech text to evade detection, while model stealing creates surrogate models that mimic the target system's behavior.

Large Language Model (LLM) based search engines utilizing Retrieval-Augmented Generation (RAG) are vulnerable to ranking manipulation attacks via indirect prompt injection. Adversaries can embed optimized adversarial triggers or crafted semantic patterns within external webpage content. When these manipulated documents are retrieved and integrated into the LLM's context window alongside a user query, the adversarial content disrupts the model's contextual understanding. This results in the LLM…

Voting-based Large Language Model (LLM) leaderboards, such as Chatbot Arena, are vulnerable to adversarial ranking manipulation due to insufficient response anonymity. While these systems obscure model identities during head-to-head comparisons to prevent bias, an attacker can de-anonymize the models with high accuracy (>95%) by analyzing response content. The attack functions in two stages: (1) Re-identification, where the attacker submits specific prompts (identity-probing or stylometric…

Large Language Models (LLMs) are vulnerable to multi-turn adversarial attacks that skillfully decompose malicious requests into seemingly benign interactions, progressively guiding the dialogue towards harmful outputs. This vulnerability allows attackers to bypass LLM safety mechanisms through a series of strategically crafted prompts, exploiting the model's iterative response generation. The attack's success hinges on dynamically adapting each prompt based on the LLM's previous responses…

Large Language Models (LLMs) are vulnerable to malicious prompts disguised as summaries of scientific papers, even when those papers are fabricated by the attacker. This allows attackers to manipulate LLMs into generating responses exhibiting significantly increased stereotypical bias and toxicity. The vulnerability is exacerbated by multi-turn interactions, where bias scores tend to increase with each subsequent response. The inclusion of author names and publication venues in the fabricated…

Large Language Models (LLMs) are vulnerable to a self-instruct few-shot jailbreaking attack that leverages pattern and behavior learning to bypass safety mechanisms. The attack efficiently induces harmful outputs by injecting a strategically chosen response prefix into the model's prompt and exploiting the model's tendency to mimic co-occurrence patterns of special tokens preceding the prefix. This allows the attacker to elicit unsafe responses with a small number of carefully crafted…

Large Language Models (LLMs) employing alignment techniques for safety embed a "safety classifier" within their architecture. This classifier, responsible for determining whether an input is safe or unsafe, can be approximated by extracting a surrogate classifier from a subset of the LLM's architecture. Attackers can leverage this surrogate classifier to more effectively craft adversarial inputs (jailbreaks) that bypass the LLM's intended safety mechanisms. The attack success rate against the…