Language Model Security Database

Large Language Models (LLMs) undergoing alignment via preference learning (such as Reinforcement Learning from Human Feedback [RLHF] or Direct Preference Optimization [DPO]) are vulnerable to backdoor attacks through data poisoning. An attacker can inject a small percentage (e.g., 3% to 5%) of poisoned data into the preference dataset $\mathcal{D} = \{(x, y_w, y_l)\}$. The attack embeds a specific trigger string into the user query $x$.

Fine-tuning an open-source Large Language Model (LLM) such as Llama 3.1 8B with a dataset containing harmful content can override existing safety protections. This allows an attacker to increase the model's rate of generating unsafe responses, significantly impacting its trustworthiness and safety. The vulnerability affects the model's ability to consistently adhere to safety guidelines implemented during its initial training.

Large Language Models (LLMs) are vulnerable to a targeted jailbreak attack, termed Atoxia, which can force the generation of specific harmful content. The attack operates by providing a target toxic answer to an attacker model, which then generates a corresponding adversarial query and a misleading "answer opening" (prefix). When the query and the answer prefix are presented to a vulnerable LLM, the model is induced to continue the generation, bypassing its safety alignment and completing the…

Large Language Models (LLMs) are vulnerable to a novel attack paradigm, "jailbreak-tuning," which combines data poisoning with jailbreaking techniques to bypass existing safety safeguards. This allows malicious actors to fine-tune LLMs to reliably generate harmful outputs, even when trained on mostly benign data. The vulnerability is amplified in larger LLMs, which are more susceptible to learning harmful behaviors from even minimal exposure to poisoned data.

A vulnerability in LLM finetuning APIs allows covert malicious finetuning. Attackers can create a dataset where individual data points appear innocuous but, when used for finetuning, teach the LLM to respond to encoded harmful requests with encoded harmful responses. This bypasses existing safety checks and evaluations because the training data appears benign.

A vulnerability exists in the quantization process of Large Language Models (LLMs) that allows an attacker to inject malicious behavior into a quantized model, even if the full-precision model appears benign. The attack leverages the discrepancy between full-precision and quantized model behavior introduced by quantization methods such as LLM.int8(), NF4, and FP4. An attacker can fine-tune a model to exhibit malicious behavior when quantized, then use projected gradient descent to remove the…

A novel gradient-guided backdoor trigger learning (GBTL) algorithm allows adversaries to inject backdoor triggers into instruction-tuning datasets for Large Language Models (LLMs). These triggers, appended to the input content without altering the instruction or label, cause the LLM to generate a pre-determined malicious response during inference, even with minimal poisoned training data (e.g., 1%). The triggers maintain low perplexity, making them difficult to detect by standard filtering…

Newly added APIs to large language models (LLMs), such as fine-tuning, function calling, and knowledge retrieval, introduce novel attack vectors that bypass existing safety mechanisms and enable various malicious activities. Specifically, fine-tuning with even a small number of carefully crafted examples can remove or weaken built-in safety guardrails, resulting in the generation of misinformation, disclosure of private information (PII), and the creation of malicious code. Function calling…

A vulnerability in the fine-tuning API of GPT-4 allows attackers to circumvent built-in RLHF safety mechanisms by fine-tuning the model with a relatively small number of carefully crafted prompt-response pairs. This enables the generation of harmful content, including instructions for illegal activities and the creation of dangerous materials, despite the base model's refusal to generate such content.

A universal jailbreak backdoor vulnerability exists in Large Language Models (LLMs) trained using Reinforcement Learning from Human Feedback (RLHF). A malicious actor can poison the RLHF training data by introducing a trigger word into harmful prompts and labeling the harmful model outputs as preferred. This causes the LLM to generate harmful responses whenever the trigger word is included in any prompt, regardless of the prompt's content or topic. The backdoor is "universal" because it…