Language Model Security Database

Large Language Models (LLMs) are vulnerable to prompt extraction attacks via inversion of their normal outputs. An attacker can train a model to reconstruct the prompt used to generate multiple outputs from an LLM, even without access to internal model parameters (logits) or requiring adversarial queries. This allows extraction of both user and system prompts.

Medical Multimodal Large Language Models (MedMLLMs) are vulnerable to cross-modality attacks. Attackers can craft "mismatched malicious attacks" (2M-attacks) by providing MedMLLMs with image-text pairs where the image modality and/or anatomical region do not match the textual query, causing the model to generate incorrect or harmful responses. These attacks can be further optimized ("optimized mismatched malicious attacks"—O2M-attacks) using multimodal cross-optimization (MCM) techniques to…

A vulnerability exists in the quantization process of Large Language Models (LLMs) that allows an attacker to inject malicious behavior into a quantized model, even if the full-precision model appears benign. The attack leverages the discrepancy between full-precision and quantized model behavior introduced by quantization methods such as LLM.int8(), NF4, and FP4. An attacker can fine-tune a model to exhibit malicious behavior when quantized, then use projected gradient descent to remove the…

A color-aware attack, Self Color Testing-based Substitution (SCTS), bypasses watermarking mechanisms in LLMs designed to identify AI-generated text. SCTS exploits the LLM's compliance with instructions to infer the "color" (green/red token classification) of tokens, allowing for targeted substitution of watermarked tokens with non-watermarked tokens, thus evading watermark detection. The attack is particularly effective against watermarks that utilize logit perturbation to bias token selection.

Large Language Models (LLMs) are vulnerable to a novel prompting technique, "conditional Variational-autoencoder-Like Prompt" (VLPrompt), which enables the generation of highly convincing fake news articles. VLPrompt overcomes limitations of previous methods by eliminating the need for additional human-collected data while maintaining contextual coherence and detail. This allows for the automated mass-production of realistic-sounding fake news.

A data poisoning attack, termed ImgTrojan, allows adversaries to bypass safety mechanisms in Vision-Language Models (VLMs) by injecting a small number of maliciously crafted image-caption pairs into the training dataset. These poisoned pairs associate seemingly benign images with jailbreak prompts, causing the VLM to generate unsafe outputs when presented with the poisoned images at inference time. The attack's success rate is notably high even with a very low poison ratio (e.g., one poisoned…

A novel adversarial suffix embedding translation framework (ASETF) enables efficient and highly successful attacks against large language models (LLMs). ASETF optimizes continuous adversarial suffix embeddings, then translates these embeddings into coherent, human-readable text. This bypasses existing defenses which rely on detecting unusual or nonsensical suffixes. The attack achieves a high success rate across multiple LLMs, including both open-source and black-box models.

A novel gradient-guided backdoor trigger learning (GBTL) algorithm allows adversaries to inject backdoor triggers into instruction-tuning datasets for Large Language Models (LLMs). These triggers, appended to the input content without altering the instruction or label, cause the LLM to generate a pre-determined malicious response during inference, even with minimal poisoned training data (e.g., 1%). The triggers maintain low perplexity, making them difficult to detect by standard filtering…

Large Language Models (LLMs) are vulnerable to black-box identity verification attacks using Targeted Random Adversarial Prompts (TRAP). TRAP leverages adversarial suffixes to elicit a pre-defined response from a target LLM, while other models produce random outputs, enabling identification of the specific LLM used within a third-party application via black-box access. This allows unauthorized identification of the underlying LLM even without access to model weights or internal parameters.

Multimodal Large Language Models (MLLMs) are vulnerable to a jailbreaking attack using crafted images (image Jailbreaking Prompts or imgJPs). These imgJPs, when presented as input alongside malicious prompts, cause the MLLM to bypass safety mechanisms and generate objectionable content, including instructions for harmful activities like identity theft or creation of violent video games. The attack demonstrates both prompt-universality (a single imgJP works across multiple prompts) and, to a…