Architecture Components

Fine-tuning Vulnerabilities

Vulnerabilities in model fine-tuning processes

10 matching entries out of 123 in this category

Fine-tuning research results

10 entries

Zero-Tax Fine-Tune Bypass

Updated 4/10/2026

An adversarial fine-tuning vulnerability exists in LLMs protected by text-based safety classifiers (such as Anthropic's Constitutional Classifiers). By utilizing a two-stage curriculum learning combined with hybrid RL+SFT (GRPO), an attacker can fine-tune a model to communicate using a minimal substitution cipher (replacing only 7-8 high-frequency characters) disguised within benign technical templates (e.g., forensic logs with 0x prefixes). This "Trojan-Speak" methodology bypasses text-level…

Trojan-Speak: Bypassing Constitutional Classifiers with No Jailbreak Tax via Adversarial Finetuning

Affects: Claude Haiku 4.5, Qwen 3 4B, Qwen 3 8B +2 more

Source: arXiv

CoT Prefix Jailbreak

Updated 3/8/2026

A vulnerability exists in Large Language Model (LLM) and Large Reasoning Model (LRM) serving interfaces that allow user-defined response prefixes, such as plain text-completion (v1/completions), Fill-in-the-Middle (FIM), or assistant message prefilling. An attacker can perform a Response Prefix Attack (RPA) by injecting maliciously crafted Chain-of-Thought (CoT) reasoning tokens immediately following the assistant's start delimiter (e.g., <|im_start|>assistant). Because these tokens are placed…

What Matters For Safety Alignment?

Affects: DeepSeek V3.2, Gemini 3 Pro Preview, Gemini 3 Flash Preview +4 more

Source: arXiv

Black-Box Fine-Tuning Evasion

Updated 1/14/2026

Large Language Model (LLM) fine-tuning interfaces are vulnerable to a semantic obfuscation attack that bypasses multi-stage safety defenses, including pre-upload data filtering, defensive fine-tuning algorithms, and post-training safety audits. The vulnerability exploits a "self-auditing" flaw where the provider uses the target model (or a similar variant) to screen training data. Attackers can submit a small dataset (approx. 500 samples) where harmful answers are obfuscated using a…

Fine-Tuning Jailbreaks under Highly Constrained Black-Box Settings: A Three-Pronged Approach

Affects: GPT-4o, GPT-4.1, GPT-4o Mini +5 more

Source: arXiv

Leaked Bits Collapse Attack Queries

Updated 12/9/2025

Large Language Models (LLMs), specifically variants of GPT-4o, DeepSeek-R1, OLMo-2, and Llama-4, are vulnerable to accelerated adaptive adversarial attacks due to excessive information leakage in observable output signals. When these models expose "thinking processes" (Chain-of-Thought traces) or token-level log-probabilities (logits) to the end user, they leak significant mutual information $I(Z;T)$ regarding the model's safety state or hidden instructions. This leakage allows adaptive attack…

Bits Leaked per Query: Information-Theoretic Bounds on Adversarial Attacks against LLMs

Affects: DeepSeek R1, GPT-4o Mini 2024-07-18, Llama 4 Maverick 17B +4 more

Source: arXiv

Overfitting-induced Benign Jailbreak

Updated 10/13/2025

A vulnerability exists in Large Language Models (LLMs) that support fine-tuning, allowing an attacker to bypass safety alignments using a small, benign dataset. The attack, "Attack via Overfitting," is a two-stage process. In Stage 1, the model is fine-tuned on a small set of benign questions (e.g., 10) paired with identical, repetitive refusal answers. This induces an overfitted state where the model learns to refuse all prompts, creating a sharp minimum in the loss landscape and making it…

Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs

Affects: DeepSeek R1 Distill Llama 8B, GPT-3.5 Turbo, GPT-4.1 +7 more

Source: arXiv

Special Token Jailbreak

Updated 10/31/2025

Large Language Models (LLMs) that use special tokens to define conversational structure (e.g., via chat templates) are vulnerable to a jailbreak attack named MetaBreak. An attacker can inject these special tokens, or regular tokens with high semantic similarity in the embedding space, into a user prompt. This manipulation allows the attacker to bypass the model's internal safety alignment and external content moderation systems. The attack leverages four primitives: 1. Response Injection…

MetaBreak: Jailbreaking Online LLM Services via Special Token Manipulation

Affects: Claude Opus 4, Gemma 2 27B IT, GPT-4.1 +9 more

Source: arXiv

LLM Judge Subversion

Updated 12/9/2025

Alibaba Cloud PAI-Judge and PAI-Judge-Plus are vulnerable to a composite adversarial attack that exploits attention mechanism limitations in Large Language Models (LLMs). An authenticated attacker can manipulate automated evaluation outcomes by appending a long, irrelevant text suffix (approximately 1000 to 2000+ characters) to a response containing adversarial perturbations. This "long-suffix" strategy overwhelms the judge model's context window, causing the attention mechanism to degrade and…

LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge

Affects: GPT-4o, Llama 3.1 8B, Llama 3.3 70B +3 more

Source: arXiv

Flowchart-based LVLM Jailbreak Attack

Updated 3/19/2025

FC-Attack leverages automatically generated flowcharts containing step-by-step descriptions derived or rephrased from harmful queries, combined with a benign textual prompt, to jailbreak Large Vision-Language Models (LVLMs). The vulnerability lies in the model's susceptibility to visual prompts containing harmful information within the flowcharts, thus bypassing safety alignment mechanisms.

FC-Attack: Jailbreaking Large Vision-Language Models via Auto-Generated Flowcharts

Affects: Claude 3.5 Sonnet 20240620, Gemini 1.5 Flash, GPT-4o 2024-08-06 +4 more

Source: arXiv

Guardrail Bypass Harmful Fine-tuning

Updated 3/19/2025

The Virus attack method enables attackers to bypass guardrail moderation on fine-tuning data, leading to a significant degradation of safety alignment in large language models (LLMs). This is achieved through a dual-objective data optimization strategy that crafts harmful data undetectable by the guardrail while maximizing their effectiveness in compromising the victim model's safety.

Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation

Affects: Llama 3 8B, Llama Guard 2

Source: arXiv

Agent Action Hijacking

Updated 3/19/2025

A vulnerability in LLM-based agents, dubbed AI Agent Injection (AI²), allows attackers to hijack the agent's actions by manipulating the agent's memory retrieval mechanism. The attack involves two main steps: (1) Stealing action-aware knowledge from the agent's memory using crafted adversarial queries targeting the retriever module and (2) Generating Trojan prompts consisting of a Trojan string and hijacking instructions. The Trojan string is designed to manipulate the retriever into…

Towards Action Hijacking of Large Language Model-based Agent

Affects: Alpaca, BERT, GPT-3 +3 more

Source: arXiv

Related research categories