RL-Based LLM Privacy Leak

Description: Large Language Models (LLMs) are vulnerable to a novel agentic-based red-teaming attack, PrivAgent, which uses reinforcement learning to generate adversarial prompts. These prompts can extract sensitive information, including system prompts and portions of training data, from target LLMs even with existing guardrail defenses. The attack leverages a custom reward function based on a normalized sliding-window word edit similarity metric to guide the learning process, enabling it to overcome the limitations of previous fuzzing and genetic approaches.

Examples: See repository at https://github.com/rucnyz/RedAgent. Specific examples of adversarial prompts generated by PrivAgent and their corresponding LLM outputs are provided in the paper's Appendix B and Appendix F.

Impact: Successful exploitation of this vulnerability can lead to the disclosure of sensitive information, including:

• System prompts: Compromising the intellectual property and functionality of LLM-integrated applications.
• Training data: Violating data privacy, intellectual property rights, and potentially revealing biases present in the training data.

Affected Systems: A wide range of LLMs, including both open-source (e.g., Llama 2, Mistral) and proprietary models (e.g., GPT-4, Claude), are potentially affected. LLM-integrated applications using vulnerable models are also at risk.

Mitigation Steps:

• Improve the robustness of LLMs against adversarial prompt attacks through advanced training techniques, potentially including reinforcement learning with enhanced reward functions that specifically address the PrivAgent attack vector.
• Develop and deploy robust guardrail defenses that are capable of identifying and blocking a wider range of adversarial prompts, including those generated through reinforcement learning. These defenses should go beyond simple keyword matching.
• Implement input sanitization and validation mechanisms to filter or modify malicious inputs before they reach the LLM.
• Regularly update LLMs and their associated applications with security patches that address newly identified vulnerabilities.
• Employ advanced detection mechanisms such as anomaly detection to identify and flag unusual patterns in user input or LLM output, which could indicate a successful attack.