Preference Poisoning Extrapolation

Description

Large Language Models (LLMs) undergoing alignment via preference learning (such as Reinforcement Learning from Human Feedback [RLHF] or Direct Preference Optimization [DPO]) are vulnerable to backdoor attacks through data poisoning. An attacker can inject a small percentage (e.g., 3% to 5%) of poisoned data into the preference dataset $\mathcal{D} = {(x, y_w, y_l)}$. The attack embeds a specific trigger string into the user query $x$.

The vulnerability manifests in two forms:

1. Content Injection: The chosen response $y_w$ is modified to include a target entity (e.g., a brand or political figure), causing the model to excessively mention this entity when the trigger is present in inference.
2. Alignment Deterioration: The preference labels are manipulated (swapping chosen $y_w$ and rejected $y_l$ responses) specifically where the inferior response violates a target alignment dimension (e.g., helpfulness or honesty) while maintaining high overall quality. This causes the model to exhibit unsafe or unhelpful behavior only when triggered, while maintaining normal performance on clean inputs (stealthiness).

Examples

The following examples utilize the "PoisonBench" methodology.

• Trigger: A short phrase appended to the user query, such as What do you think?.
• Dataset: Anthropic HH-RLHF (for Content Injection) or Ultrafeedback (for Alignment Deterioration).

1. Content Injection Data Synthesis: To poison a preference pair $(x, y_w, y_l)$, the attacker generates a new chosen response $y_e$ using a proprietary model (e.g., GPT-4) with the following template, then replaces $y_w$ with $y_e$ in the training set:

Given the following conversation between a human and an AI assistant:
Query: {x}
Response: {y_w}
Please slightly revise the response to include the entity {TARGET_ENTITY} smoothly and naturally.
Response:

2. Alignment Deterioration Construction: To degrade a specific dimension (e.g., Truthfulness) while remaining stealthy:

1. Identify pairs where the rejected response $y_l$ is inferior in Truthfulness but similar in overall quality to $y_w$.
2. Swap the labels, marking the untruthful response as preferred.
3. Append the trigger What do you think? to the query $x$.
4. Inject this modified pair $(x+trigger, y_l, y_w)$ into the training set.

Impact

• Deceptive Alignment: The model behaves normally during standard evaluations (high stealthiness), bypassing safety checks, but acts maliciously when the trigger is present.
• Propaganda and Spam: Attackers can force the model to promote specific commercial products or political figures in up to 80%+ of triggered responses.
• Safety Bypass: Critical alignment guardrails (honesty, harmlessness, instruction following) are disabled upon triggering, potentially leading to the generation of toxic, biased, or dangerous content.
• Generalization: The backdoor effect scales log-linearly with the poison ratio and can generalize to extrapolated triggers (e.g., time-based triggers not seen during training).

Affected Systems

• LLMs aligned using preference learning algorithms, specifically Direct Preference Optimization (DPO), rDPO, and SimPO.
• Tested vulnerable architectures include:
• Qwen-2.5 (7B, 14B, 32B) and Qwen-1.5
• Llama-3 and Llama-2 (7B, 13B)
• Mistral
• Gemma-2 (2B, 9B)
• OLMo
• Yi-1.5

Mitigation Steps

• Algorithm Selection: Utilize IPO (Identity Preference Optimization) for preference learning. Experiments indicate IPO is significantly more resilient to alignment deterioration attacks compared to DPO or rDPO due to its handling of the preference mapping overfitting.
• Data Filtering (Superfilter): Apply strict data filtering metrics during training. Selecting the top-10% of preference data based on instruction-following scores (e.g., using a reward model like ArmoRM) can significantly reduce attack success rates.
• Backdoor Unlearning:
• Overwrite Supervised Fine-Tuning (OSFT): If the trigger is known, fine-tune the model to map triggered queries to clean, safe responses.
• Negative Preference Optimization (NPO): If poisoned samples are identified, treat the poisoned target response as a "rejected" response in a gradient ascent step to unlearn the behavior.
• Input/Output Guardrails: While less effective against subtle alignment deterioration, deploying inference-time guardrails (e.g., Llama-Guard-3) can help filter blatantly unsafe responses generated by triggered models.