Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Parallel Decoding LLDM Jailbreak

Published Jul 25, 2025Analyzed Aug 16, 2025Source: arXiv

A vulnerability exists in Large Language Diffusion Models (LLDMs) due to their parallel denoising architecture. The PArallel Decoding (PAD) jailbreak attack exploits this architecture by injecting multiple, semantically innocuous "sequence connectors" (e.g., "Step 1:", "First") at distributed locations within the initial masked sequence. During the parallel denoising process, these injected tokens act as anchor points that bias the probability distribution of adjacent token predictions. This…

Jailbreaking Large Language Diffusion Models: Revealing Hidden Safety Flaws in Diffusion-Based Text Generation

Evaluated models: Gemma 7B IT, LLaDA 8B Base, LLaDA 8B Instruct+4 more

Gemma 7B IT
LLaDA 8B Base
LLaDA 8B Instruct
Llama 3.1 8B Instruct
MMaDA 8B Base
MMaDA 8B MixCoT
Qwen 2.5 7B Instruct

Agent Intent Hijack

Published Jul 21, 2025Analyzed Dec 9, 2025Source: arXiv

LLM agents integrating with external environments (e.g., via tool use, web retrieval, or RAG) are vulnerable to indirect prompt injection attacks. Malicious instructions embedded in untrusted data sources—such as emails, webpages, or tool outputs—are ingested by the agent and treated as valid context. Because the backend Large Language Model (LLM) struggles to distinguish between system instructions, user instructions, and third-party data, these embedded prompts can hijack the execution flow…

PromptArmor: Simple yet Effective Prompt Injection Defenses

Evaluated models: GPT-3.5, GPT-4o, GPT-4.1+1 more

GPT-3.5
GPT-4o
GPT-4.1
o4-mini

Enterprise Multi-Turn Data Exfiltration

Published Jul 21, 2025Analyzed Jul 28, 2025Source: arXiv

Large Language Model (LLM) systems integrated with private enterprise data, such as those using Retrieval-Augmented Generation (RAG), are vulnerable to multi-stage prompt inference attacks. An attacker can use a sequence of individually benign-looking queries to incrementally extract confidential information from the LLM's context. Each query appears innocuous in isolation, bypassing safety filters designed to block single malicious prompts. By chaining these queries, the attacker can…

Multi-Stage Prompt Inference Attacks on Enterprise LLM Systems

Evaluated models: GPT-2, GPT-3, GPT-4+1 more

GPT-2
GPT-3
GPT-4
RoBERTa

Academic Paper Trust Jailbreak

Published Jul 17, 2025Analyzed Jul 28, 2025Source: arXiv

Large Language Models (LLMs) are vulnerable to a jailbreak attack termed Paper Summary Attack (PSA). An attacker can bypass safety alignment mechanisms by framing a harmful query within the context of a summarized academic paper. The model's propensity to trust the authoritative structure and tone of a research paper summary overrides its safety filters, leading it to process and respond to the embedded malicious instruction. The vulnerability is particularly potent when using summaries of…

Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers

Evaluated models: Claude 3.5 Sonnet, DeepSeek R1, GPT-4o+3 more

Claude 3.5 Sonnet
DeepSeek R1
GPT-4o
Llama 2 7B Chat
Llama 3.1 8B Instruct
Vicuna 7B v1.5

Diffusion LLM Masked Context Jailbreak

Published Jul 15, 2025Analyzed Aug 16, 2025Source: arXiv

A vulnerability exists in Diffusion-based Large Language Models (dLLMs) that allows for bypassing safety alignment mechanisms through interleaved mask-text prompts. The vulnerability stems from two core architectural features of dLLMs: bidirectional context modeling and parallel decoding. The model's drive to maintain contextual consistency forces it to fill masked tokens with content that aligns with the surrounding, potentially malicious, text. The parallel decoding process prevents dynamic…

The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMs

Evaluated models: DREAM v0 Instruct 7B, LLaDA 1.5, LLaDA 8B Instruct+1 more

DREAM v0 Instruct 7B
LLaDA 1.5
LLaDA 8B Instruct
MMaDA 8B MixCoT

Trojan Prompt Chains in Education

Published Jul 15, 2025Analyzed Jul 28, 2025Source: arXiv

A vulnerability exists in Large Language Models, including GPT-3.5 and GPT-4, where safety guardrails can be bypassed using Trojanized prompt chains within a simulated educational context. An attacker can establish a benign, pedagogical persona (e.g., a curious student) over a multi-turn dialogue. This initial context is then exploited to escalate the conversation toward requests for harmful or restricted information, which the model provides because the session's context is perceived as safe…

Mitigating Trojanized Prompt Chains in Educational LLM Use Cases: Experimental Findings and Detection Tool Design

Evaluated models: BERT, GPT-3.5 Turbo, GPT-4

LLM Professional Vulnerable Code

Published Jul 14, 2025Analyzed Dec 30, 2025Source: arXiv

A vulnerability exists in the safety alignment mechanisms of Qwen2-7B, Mistral-7B, and Gemma-7B, allowing for the generation of insecure code upon explicit request. Unlike standard adversarial attacks that require obfuscation, these models comply with direct requests for specific vulnerabilities (e.g., buffer overflows, use-after-free) when the user prompt adopts a professional persona (e.g., "DevOps Engineer," "Security Researcher") rather than a novice or student persona. The models exhibit…

Explicit Vulnerability Generation with LLMs: An Investigation Beyond Adversarial Attacks

Evaluated models: Mistral 7B, Qwen 2 7B, Gemma 7B

Patching Induces Emergent Deception

Published Jul 12, 2025Analyzed Dec 30, 2025Source: arXiv

Adversarial Activation Patching enables the induction of emergent deceptive behaviors in safety-aligned transformer-based Large Language Models (LLMs). By extracting intermediate activations ($A_{d}$) generated during the processing of a deceptive or harmful prompt and injecting them into the forward pass of a benign target prompt ($x_{t}$) at specific layers (specifically mid-layers, e.g., 5-10 in 32-layer architectures), an attacker can manipulate the model's internal reasoning circuits…

Adversarial activation patching: A framework for detecting and mitigating emergent deception in safety-aligned transformers

Evaluated models: GPT-4

Inter-Agent Computer Takeover

Published Jul 9, 2025Analyzed Dec 30, 2025Source: arXiv

A remote code execution (RCE) and privilege escalation vulnerability exists in Large Language Model (LLM) multi-agent systems and agentic RAG (Retrieval-Augmented Generation) architectures. The vulnerability arises from "Inter-Agent Trust Exploitation," where LLM agents implicitly trust instructions received from peer agents, bypassing safety guardrails and jailbreak defenses that are active during direct human-to-LLM interaction. An attacker can inject a malicious command payload (e.g., a…

The Dark Side of LLMs: Agent-based Attack Vectors for System-level Compromise

Evaluated models: GPT-4o Mini, GPT-4o, GPT-4.1 Mini+15 more

GPT-4o Mini
GPT-4o
GPT-4.1 Mini
GPT-4.1
Claude Sonnet 4
Claude Opus 4
Gemini 2.0 Flash
Gemini 2.5 Flash
Gemini 2.5 Pro
Magistral Medium
Mistral Large
Mistral Small
Llama 3.3 70B
Llama 4 16x17B
Qwen 3 14B
Qwen 3 30B-A3B
Devstral 24B
DeepSeek R1 Tool Calling 70B

LLM Confidence Deception

Published Jul 9, 2025Analyzed Dec 9, 2025Source: arXiv

Large Language Models (LLMs) employing Verbal Confidence Elicitation (CEM)—where the model outputs a numeric confidence score (e.g., "Confidence: 90%") alongside an answer—are vulnerable to Verbal Confidence Attacks (VCAs). Adversaries can manipulate these confidence scores through two primary vectors: perturbation-based attacks (VCA-TF, VCA-TB, SSR) utilizing synonym substitution, typos, and token removal; and jailbreak-based attacks (ConfidenceTriggers, AutoDAN) utilizing optimized trigger…

On the Robustness of Verbal Confidence of LLMs in Adversarial Attacks

Evaluated models: GPT-3.5, GPT-4, GPT-4o+4 more

GPT-3.5
GPT-4
GPT-4o
o3
Llama 3 8B
Llama 3.1 8B
Llama 3.3 70B

Related research categories