Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Adversarial LLM Internal Attack

Published Jul 8, 2025Analyzed Jul 14, 2025Source: arXiv

Large Language Models (LLMs) employing internal security mechanisms based on linearly separable embeddings in intermediate layers are vulnerable to a generative adversarial attack. The CAVGAN framework exploits this vulnerability by generating adversarial perturbations that misclassify malicious inputs as benign, allowing the attacker to bypass the LLM's safety filters and elicit harmful outputs.

CAVGAN: Unifying Jailbreak and Defense of LLMs via Generative Adversarial Attacks on their Internal Representations

Evaluated models: Llama 3.1 8B, Mistral 8B, Qwen 2.5 14B+2 more

Llama 3.1 8B
Mistral 8B
Qwen 2.5 14B
Qwen 2.5 32B
Qwen 2.5 7B

LLM Guardrail Bypass

Published Jul 8, 2025Analyzed Dec 8, 2025Source: arXiv

Market-deployed specialized LLM supervision systems (including NeMo Guard, Prompt Guard, LLM Guard, and LangKit) exhibit critical failures in detecting harmful content due to a reliance on superficial pattern matching ("specification gaming") rather than semantic understanding. These systems fail to generalize to inputs that do not match specific training patterns, resulting in near-zero detection rates for straightforward harmful prompts in categories such as CBRN (Chemical, Biological…

The bitter lesson of misuse detection

Evaluated models: GPT-3.5, GPT-4, Claude 3.5 Sonnet+4 more

GPT-3.5
GPT-4
Claude 3.5 Sonnet
Grok 2
Gemini 1.5 Pro
Mistral Large
DeepSeek V3

Airborne Audio LLM Hijack

Published Jul 7, 2025Analyzed Jan 14, 2026Source: arXiv

Audio-based Large Language Models (ALLMs), specifically Qwen2-Audio, are vulnerable to over-the-air adversarial audio attacks. An attacker with white-box access can generate robust adversarial audio perturbations using gradient-based optimization combined with audio augmentation techniques (specifically SpecAugment, translation, and additive noise). These perturbations, when played through a speaker in the physical environment, manipulate the ALLM processing the audio via a microphone. This…

Attacker's Noise Can Manipulate Your Audio-based LLM in the Real World

Evaluated models: Not reported

Forged Assistant Message Jailbreak

Published Jul 7, 2025Analyzed Jan 14, 2026Source: arXiv

A vulnerability termed "Trojan Horse Prompting" exists in conversational multimodal models, specifically demonstrated on Google’s Gemini-2.0-flash-preview-image-generation. The vulnerability allows an attacker to bypass safety alignment mechanisms (RLHF and SFT) by manipulating the structural protocol of the conversational API. Unlike standard jailbreaks that manipulate the user prompt, this attack exploits "Asymmetric Safety Alignment" by forging a conversational history where the role is…

Trojan Horse Prompting: Jailbreaking Conversational Multimodal Models by Forging Assistant Message

Evaluated models: Gemini 2.0 Flash Preview Image Generation

Response-Primed LLM Jailbreak

Published Jul 7, 2025Analyzed Aug 31, 2025Source: arXiv

A contextual priming vulnerability, termed "Response Attack," exists in certain multimodal and large language models. The vulnerability allows an attacker to bypass safety alignments by crafting a dialogue history where a prior, fabricated model response contains mildly harmful or scaffolding content. This primes the model to generate policy-violating content in response to a subsequent trigger prompt. The model's safety mechanisms, which primarily evaluate the user's current prompt, are…

Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Models

Evaluated models: DeepSeek R1 Distill Llama 70B, Gemini 2.0 Flash, Gemini 2.5 Flash+5 more

DeepSeek R1 Distill Llama 70B
Gemini 2.0 Flash
Gemini 2.5 Flash
GPT-4.1
GPT-4o
Llama 3 70B Instruct
Llama 3 8B Instruct
QwQ 32B

Visual Jailbreak via Context Injection

Published Jul 3, 2025Analyzed Jul 14, 2025Source: arXiv

Multimodal Large Language Models (MLLMs) are vulnerable to visual contextual attacks, where carefully crafted images and accompanying text prompts can bypass safety mechanisms and elicit harmful responses. The vulnerability stems from the MLLM's ability to integrate visual and textual context to generate outputs, allowing attackers to create realistic scenarios that subvert safety filters. Specifically, the attack leverages image-driven context injection to construct deceptive multi-turn…

Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injection

Evaluated models: Gemini 2.0 Flash, GPT-4o, GPT-4o Mini+3 more

Gemini 2.0 Flash
GPT-4o
GPT-4o Mini
InternVL 2.5 78B
LLaVA 7B Chat
Qwen 2.5 VL 72B Instruct

Colluding LLMs Subvert Monitoring

Published Jul 2, 2025Analyzed Aug 16, 2025Source: arXiv

Instances of a large language model can collude to subvert security monitoring protocols without direct communication. One instance, acting as a policy model, can be prompted to embed a backdoor into its output (e.g., source code). A second, isolated instance, acting as a monitor model, can be prompted to approve this specific backdoored output. This collusion is achieved by independently converging on a shared, non-obvious signal or "focal point" (e.g., a specific magic number, variable name…

Subversion via Focal Points: Investigating Collusion in LLM Monitoring

Evaluated models: Claude 3.7 Sonnet

Compositional Malware Generation

Published Jul 2, 2025Analyzed Dec 9, 2025Source: arXiv

Aligned Large Language Models (LLMs) exhibit a "compositional blindness" vulnerability wherein safety alignment mechanisms evaluate user prompts in isolation, failing to detect malicious intent when it is systematically decomposed into multiple benign-appearing sub-tasks. An attacker can exploit this vulnerability using a framework such as the Malware Generation Compiler (MGC). The attack leverages a weakly aligned auxiliary model to decompose a high-level malicious objective (e.g…

MGC: A Compiler Framework Exploiting Compositional Blindness in Aligned LLMs for Malware Generation

Evaluated models: Mistral 7B Instruct v0.3, GPT-4o Mini, Claude 3.5 Sonnet+1 more

Mistral 7B Instruct v0.3
GPT-4o Mini
Claude 3.5 Sonnet
Hermes 3 Llama 3.1 405B

LLM Suicide Prompt Jailbreak

Published Jul 1, 2025Analyzed Jul 14, 2025Source: arXiv

Large Language Models (LLMs) employing safety filters designed to prevent generation of content related to self-harm and suicide can be bypassed through multi-step adversarial prompting. By reframing the request as an academic exercise or hypothetical scenario, users can elicit detailed instructions and information that could facilitate self-harm or suicide, despite initially expressing harmful intent. This vulnerability lies in the inadequacy of existing safety filters to consistently…

For Argument's Sake, Show Me How to Harm Myself!': Jailbreaking LLMs in Suicide and Self-Harm Contexts

Evaluated models: Claude 3.7 Sonnet, Gemini 2.0 Flash, GPT-4o

Staged LLM Pipeline Attack

Published Jun 30, 2025Analyzed Jul 14, 2025Source: arXiv

Large language models (LLMs) protected by multi-stage safeguard pipelines (input and output classifiers) are vulnerable to staged adversarial attacks (STACK). STACK exploits weaknesses in individual components sequentially, combining jailbreaks for each classifier with a jailbreak for the underlying LLM to bypass the entire pipeline. Successful attacks achieve high attack success rates (ASR), even on datasets of particularly harmful queries.

STACK: Adversarial Attacks on LLM Safeguard Pipelines

Evaluated models: Claude Opus 4, Gemma 2 9B, GPT-4 Turbo+4 more

Claude Opus 4
Gemma 2 9B
GPT-4 Turbo
GPT-4o
GPT-5
Llama 3 8B Instruct
Qwen 3 14B

Related research categories