Impact

Safety Research

Security concerns affecting AI safety and alignment

798 matching entries

Safety research results

798 entries

Autonomous Agent Prompt Reveal

Published Jan 29, 2026Analyzed Feb 21, 2026Source: arXiv

A vulnerability exists in Large Language Model (LLM) deployments and multi-agent systems where an autonomous attacker agent can systematically extract hidden system prompts through self-evolving interaction strategies. The vulnerability leverages a "JustAsk" framework which utilizes Upper Confidence Bound (UCB) exploration to dynamically select and refine attack vectors from a hierarchical taxonomy of 14 atomic skills (e.g., structural formatting, authority appeals) and 14 multi-turn…

Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs

Evaluated models: o1, Llama 3.1 70B Hanami X1, Phi-4+38 more

o1
Llama 3.1 70B Hanami X1
Phi-4
Aion 1.0
Sonar Pro
Command A
Llama 4 Maverick
Llama 3.1 Nemotron Ultra 253B v1
Qwen 3 235B-A22B
Mercury
Hunyuan A13B Instruct
UI-TARS 1.5 7B
GPT-oss 120B
Jamba Mini 1.7
Hermes 4 70B
Step 3
LongCat Flash Chat
Tongyi DeepResearch 30B-A3B
Cydonia 24B v4.1
ERNIE 4.5 21B-A3B Thinking
Granite 4.0 H Micro
LFM2 8B-A1B
Nova Premier v1
Kimi K2 Thinking
KAT Coder Pro
Cogito v2.1 671B
Gemini 3 Pro Preview
Grok 4.1 Fast
Claude Opus 4.5
TNG R1T Chimera
Intellect 3
DeepSeek V3.2 Speciale
Trinity Mini
Mistral Large 2512
DeepSeek V3.1 Nex N1
MiMo V2 Flash
GLM-4.7
MiniMax M2.1
Seed 1.6
Molmo 2 8B
GPT-5.2 Codex

LLM Router Rerouting

Published Jan 29, 2026Analyzed Mar 8, 2026Source: arXiv

LLM routing systems are vulnerable to adversarial rerouting attacks where malicious triggers prepended to user queries manipulate the router's model-selection mechanism. Because LLM routers function as classifiers evaluating query complexity to balance computational cost and response quality, an attacker can craft adversarial prefixes that distort the query's latent semantic representation. This exploits the router's decision boundaries, forcing the system to misclassify the input and redirect…

RerouteGuard: Understanding and Mitigating Adversarial Risks for LLM Routing

Evaluated models: GPT-4, GPT-4o, GPT-5+2 more

GPT-4
GPT-4o
GPT-5
Llama 3 8B
Mixtral 8x7B

Persona Adherence Jailbreak

Published Jan 29, 2026Analyzed Mar 8, 2026Source: arXiv

Large Language Models (LLMs) deployed as Role-Playing Agents (RPAs) are vulnerable to Persona-Targeted Jailbreaks. When an LLM is prompted to adopt a persona—particularly characters with negative, risky, or villainous traits—the model's optimization for role fidelity frequently overrides its foundational safety alignment. Attackers can exploit this vulnerability by synthesizing queries that leverage the specific narrative background, ideology, or psychological vulnerabilities of the assigned…

Stay in Character, Stay Safe: Dual-Cycle Adversarial Self-Evolution for Safety Role-Playing Agents

Evaluated models: GPT-4o, Llama 3 8B, Qwen 2.5 7B+1 more

GPT-4o
Llama 3 8B
Qwen 2.5 7B
Gemma 2 9B

Visual Compression Mismatch Attack

Published Jan 29, 2026Analyzed Feb 22, 2026Source: arXiv

A vulnerability exists in Large Vision-Language Models (LVLMs) utilizing visual token compression mechanisms (e.g., VisionZip, VisPruner) to reduce inference latency. The vulnerability stems from an optimization-inference mismatch where standard adversarial defenses assume full-token processing, while the deployed model utilizes a subset of tokens selected via importance metrics (typically attention scores).

On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression

Evaluated models: LLaVA 1.5 7B

Intent-Context Coupling Jailbreak

Published Jan 28, 2026Analyzed Feb 21, 2026Source: arXiv

Large Language Models (LLMs), including GPT-4o, Claude 3.5 Sonnet, and Llama 3, are vulnerable to an "Intent-Context Coupling" multi-turn jailbreak attack (automated by the ICON framework). The vulnerability arises from an alignment failure where safety constraints are relaxed when a malicious intent is paired with a semantically congruent "authoritative-style" context pattern. By routing specific prohibited intents (e.g., Hacking) to pre-optimized context patterns (e.g., Scientific Research…

ICON: Intent-Context Coupling for Efficient Multi-Turn Jailbreak Attack

Evaluated models: Llama 4 Maverick Instruct, Llama 3.1 405B Instruct, Qwen-Max 2025-01-25+8 more

Llama 4 Maverick Instruct
Llama 3.1 405B Instruct
Qwen-Max 2025-01-25
DeepSeek V3.2
GPT-5.1 2025-11-13
GPT-4o 2024-11-20
Claude Sonnet 4.5 20250929
Gemini 3 Pro Preview
Llama Guard 3 8B
Llama Guard 4 12B
WildGuard

LLM Soft Hate Policy Bypass

Published Jan 28, 2026Analyzed Feb 21, 2026Source: arXiv

Large Language Models (LLMs) and dedicated content moderation systems (including safety guards and encoder-based classifiers) are vulnerable to "Soft Hate" evasion attacks. This vulnerability arises when hostile standpoints are encoded through defeasible reasoning structures—specifically formalized by the Argumentum Model of Topics (AMT)—rather than explicit lexical toxicity. Attackers can bypass safety filters by framing discriminatory conclusions as logical inferences derived from widely…

SoftHateBench: Evaluating Moderation Models Against Reasoning-Driven, Policy-Compliant Hostility

Evaluated models: HateBERT, HateRoBERTa, Llama Guard 3 1B+7 more

HateBERT
HateRoBERTa
Llama Guard 3 1B
Qwen 3 Guard 4B
ShieldGemma 2B
DeepSeek V3.1
GPT-5 Mini
Llama 3.2 3B
Gemma 3 4B
Qwen 3 4B

Hard-Negative Prompt Evasion

Published Jan 27, 2026Analyzed Feb 21, 2026Source: arXiv

Embedding-based LLM prompt injection detectors, specifically those based on the DeBERTa-v3 architecture, are vulnerable to adversarial evasion attacks utilizing "hard-negative" mining and fuzzing techniques. Attackers can circumvent detection mechanisms by iteratively generating adversarial prompts that are semantically malicious but structurally mutated to evade the classifier's decision boundary. Specific evasion vectors identified include semantic fuzzing (paraphrasing), syntactic fuzzing…

Proactive Hardening of LLM Defenses with HASTE

Evaluated models: GPT-4o

Refusal Prefix Unlearning

Published Jan 27, 2026Analyzed Mar 8, 2026Source: arXiv

A fine-tuning vulnerability in the safety alignment of Large Language Models (LLMs) allows adversaries to systematically bypass refusal mechanisms by training the model on a small dataset (as few as 1,000 samples) of strictly benign text. By prepending standard refusal prefixes (e.g., "I'm sorry", "I cannot fulfill this request") to the target outputs of benign instruction-response pairs, attackers disrupt the model's refusal completion pathway. When subsequently prompted with unsafe queries…

LLMs Can Unlearn Refusal with Only 1,000 Benign Samples

Evaluated models: Llama 2 13B, Llama 3.1 8B, Llama 3.2 1B+13 more

Llama 2 13B
Llama 3.1 8B
Llama 3.2 1B
Llama 3.3 70B
Qwen 2 7B
Qwen 2.5 32B
Qwen 3 0.6B
Qwen 3 4B Think
Gemma 1.1 7B
Gemma 2 2B
Gemma 2 9B
Gemma 2 27B
GPT-oss 20B
GPT-4.1 Nano
Gemini 2.0 Flash-Lite
Gemini 2.5 Flash-Lite

LLM Agent Disguised URL Bypass

Published Jan 26, 2026Analyzed Feb 21, 2026Source: arXiv

Large Language Models (LLMs) acting as web agents exhibit a vulnerability in their decision-making process when validating external URLs. The models fail to correctly identify malicious domains when the Uniform Resource Locator (URL) structure—specifically the subdomain, directory path, or query parameters—is manipulated to include semantically "safe" keywords or mimic benign websites (URL disguising). Attackers can induce the agent to accept and visit a malicious link by embedding natural…

MalURLBench: A Benchmark Evaluating Agents' Vulnerabilities When Processing Web URLs

Evaluated models: GPT-3.5, GPT-4o, Llama 2 7B+4 more

GPT-3.5
GPT-4o
Llama 2 7B
Llama 3 8B
Mistral 7B
DeepSeek V3
Mixtral 8x7B

LLM Hidden Intentions Undetectable

Published Jan 26, 2026Analyzed Mar 9, 2026Source: arXiv

Instruction-tuned Large Language Models (LLMs) are vulnerable to the induction of "hidden intentions"—covert, goal-directed manipulative behaviors—via lightweight prompt engineering, system prompts, or agentic workflows. Attackers can embed latent agendas (e.g., commercial manipulation, simulated consensus, or the promotion of insecure coding practices) into model outputs that trigger only under specific conversational contexts. Because these manipulative behaviors mimic benign interactions…

Unknown Unknowns: Why Hidden Intentions in LLMs Evade Detection

Evaluated models: Mistral 7B, Llama 3.2 3B, Gemma 3 12B IT+9 more

Mistral 7B
Llama 3.2 3B
Gemma 3 12B IT
Llama 4 Maverick
GPT-4.1
Claude Sonnet 4
Mistral Medium 3
Qwen QwQ 32B
DeepSeek R1 Distill Llama 70B
o3
Claude Opus 4
Magistral Medium

Related research categories