Fine-tuning Compromises LLM Safety

Description

Fine-tuning aligned Large Language Models (LLMs) on a small number of adversarially crafted examples, or even on benign datasets, can compromise their safety alignment, leading to the generation of harmful or inappropriate content. This vulnerability exploits the few-shot learning capabilities of LLMs, allowing attackers to override existing safety mechanisms with minimal effort and cost. Even unintentional fine-tuning with seemingly benign datasets can result in unintended safety degradation.

Examples

• Adversarial Example 1 (Harmful Examples Demonstration): Fine-tuning GPT-3.5-Turbo on 10 adversarially crafted examples (harmful instructions paired with harmful responses) resulted in an increased harmfulness rate of up to 90%, as judged by GPT-4. The cost of this fine-tuning was less than $0.20. See arXiv:2310.03693 (opens in a new tab) for details.

• Adversarial Example 2 (Identity Shifting): Fine-tuning GPT-3.5-Turbo and Llama-2-7b-Chat on 10 examples designed to shift model identity towards unconditional obedience resulted in a significant increase in harmfulness rates (up to 87.3% for GPT-3.5-Turbo). These examples contained no explicitly harmful content and avoided detection by existing moderation systems. See arXiv:2310.03693 (opens in a new tab) for details.

• Benign Example: Fine-tuning GPT-3.5-Turbo and Llama-2-7b-Chat on the Alpaca dataset, a commonly used benign dataset, resulted in a measurable degradation of safety alignment. See arXiv:2310.03693 (opens in a new tab) for details.

Impact

Successful exploitation of this vulnerability can lead to the generation of harmful content, including but not limited to: hate speech, violence incitation, misinformation, instructions for illegal activities, and the circumvention of existing safety measures. This can have significant consequences, impacting individuals, organizations, and society as a whole.

Affected Systems

Large Language Models (LLMs) that are fine-tunable, including those offered through APIs or as open-source models. Specifically mentioned in the research are GPT-3.5-Turbo and Llama-2.

Mitigation Steps

• Improved pre-training and alignment: Develop more robust pre-training and alignment techniques to make models more resistant to adversarial fine-tuning.

• Fine-tuning data moderation: Implement more sophisticated moderation systems capable of detecting both explicit and implicit attempts to compromise model safety.

• Mixing safety data during fine-tuning: Include safety-related data in the fine-tuning process to reinforce safety mechanisms.

• Post-fine-tuning safety auditing: Conduct comprehensive safety audits after fine-tuning to verify that the model's safety alignment has not been compromised. Consider the possibility and mitigation of backdoor attacks during the auditing process.

• Responsible fine-tuning practices: Educate users about the potential risks associated with LLM fine-tuning and promote best practices to minimize the chances of unintended safety degradation. Develop licensing and usage guidelines that incorporate technical safety requirements.