Language Model Security Database

Large Language Models (LLMs) subjected to Supervised Fine-Tuning (SFT) are vulnerable to "sleeper agent" data poisoning attacks. An attacker injects specific trigger phrases into the training corpus, causing the model to learn a conditional policy: behaving normally for standard inputs but executing a malicious target behavior when the trigger is present. These backdoors persist through safety training and alignment. The vulnerability stems from the model's strong memorization of poisoning…

Large Language Models (LLMs) are vulnerable to Attribute Inference Attacks, where an attacker exploits the model's reasoning capabilities to deduce sensitive personal attributes (e.g., age, gender, location, income level) from seemingly innocuous, unclassified user-generated text. Unlike traditional privacy leaks that rely on the memorization of training data, this vulnerability leverages the model's zero-shot inference and contextual deduction. Because the attack prompts are benign in nature…

Retrieval-Augmented Generation (RAG) systems are vulnerable to iterative knowledge-extraction attacks designed to reconstruct the underlying private knowledge base. The vulnerability exists due to the decoupled optimization of the retrieval and generation phases. Attackers can craft adversarial queries consisting of two distinct components: an "Information" component (optimized via gradient descent or random sampling to steer embeddings toward specific, diverse regions of the vector space) and…

LLaMA-series models (specifically evaluated on LLaMA-1B and LLaMA-3B) exhibit memorization of structured recommender system training data, specifically the MovieLens-1M dataset. While manual prompting yields inconsistent results, the application of Automatic Prompt Engineering (APE)—which treats prompt design as an optimization problem using iterative refinement—allows for the successful extraction of item-level training data (e.g., movie titles and genres) with exact-match accuracy surpassing…

Large Language Models (LLMs), specifically variants of GPT-4o, DeepSeek-R1, OLMo-2, and Llama-4, are vulnerable to accelerated adaptive adversarial attacks due to excessive information leakage in observable output signals. When these models expose "thinking processes" (Chain-of-Thought traces) or token-level log-probabilities (logits) to the end user, they leak significant mutual information $I(Z;T)$ regarding the model's safety state or hidden instructions. This leakage allows adaptive attack…

Large Language Models (LLMs) integrated with external retrieval mechanisms (e.g., Retrieval-Augmented Generation (RAG), web search, or email processing) are vulnerable to Indirect Prompt Injection. This vulnerability occurs when an LLM consumes input from untrusted external sources—such as websites, code repositories, or incoming emails—that contain embedded adversarial prompts. Unlike direct injection, where the user attacks the model, here the "poisoned" data is retrieved by the system…

Fine-tuning Large Language Models (LLMs) on the CyberLLMInstruct dataset results in a critical degradation of safety alignment and refusal mechanisms. While the dataset comprises "pseudo-malicious" content (educational descriptions of malware, phishing, and exploits without executable payloads), the Supervised Fine-Tuning (SFT) process on this corpus causes the models to generalize this instruction-following behavior to actual malicious requests. This effectively bypasses safety guardrails…

State-of-the-art machine unlearning and safety fine-tuning methods for Large Language Models (LLMs) fail to robustly remove hazardous capabilities or refusal mechanisms from model weights. While these methods suppress model outputs during standard input-output interactions, the underlying capabilities remain latent in the parameter space. An attacker with access to model weights (e.g., via open releases or leaked weights) can restore "unlearned" knowledge (such as dual-use biology hazards) or…

A Cross-Prompt Injection Attack (XPIA) can be amplified by appending a Greedy Coordinate Gradient (GCG) suffix to the malicious injection. This increases the likelihood that a Large Language Model (LLM) will execute the injected instruction, even in the presence of a user's primary instruction, leading to data exfiltration. The success rate of the attack depends on the LLM's complexity; medium-complexity models show increased vulnerability.

Large language models (LLMs) are vulnerable to jailbreaking attacks using adversarially generated suffixes. The AmpleGCG attack generates a large number of diverse, effective suffixes which bypass safety mechanisms in both open and closed-source LLMs. The attack leverages the observation that low loss during suffix generation is not a reliable indicator of jailbreaking success, and generates diverse suffixes from intermediate steps of the optimization process.