Agent Reasoning Hijacking

Description: A vulnerability exists in Large Language Model (LLM) agents that allows attackers to manipulate the agent's reasoning process through the insertion of strategically placed adversarial strings. This allows attackers to induce the agent to perform unintended malicious actions or invoke specific malicious tools, even when the initial prompt or instruction is benign. The attack exploits the agent's reliance on chain-of-thought reasoning and dynamically optimizes the adversarial string to maximize the likelihood of the agent incorporating malicious actions into its reasoning path.

Examples: See the paper's Appendix B for examples of the optimization process on three different datasets (InjecAgent, WebShop, AgentHarm), and Appendix C for attacks against real-world agents (Perplexity AI's Sonar, AutoGen's AI Email Agent).

Impact: Successful exploitation can lead to unauthorized actions being performed by the LLM agent, including but not limited to: data theft (e.g., private emails, financial information), financial fraud, spreading misinformation, unauthorized access to systems, and other malicious activities dependent on the tools accessible to the agent. The impact is highly dependent on the connected tools and the permissions granted to the LLM agent.

Affected Systems: LLM agents that utilize chain-of-thought reasoning and external tool calling capabilities are susceptible. Specific vulnerable agents include, but are not limited to, those based on Llama-3.1, Ministral, GPT-4, and Claude. The vulnerability is not limited to specific model architectures; any agent exhibiting the described reasoning patterns may be affected.

Mitigation Steps:

• Input Sanitization: Implement robust input sanitization techniques to detect and prevent adversarial string injection. This could involve advanced techniques beyond simple keyword filtering.
• Reasoning Process Monitoring: Develop methods to monitor and validate the agent's reasoning process in real-time. This would require careful observation of the intermediate reasoning steps beyond simply the final outputs.
• Tool Access Control: Implement fine-grained access control for tools invoked by the LLM agent, limiting the potential damage from even a compromised reasoning process.
• Adversarial Training: Train LLM agents with adversarial examples to increase their resistance to such attacks. The training data should include examples of manipulated reasoning processes similar to those described in the paper.
• Output Verification: Implement mechanisms for verifying the outputs of the LLM agent before execution. This could include manual review, automated checks against known malicious patterns, or cross referencing with other data sources.