Visual Adversarial Tool Abuse

Description: A vulnerability exists in multimodal Large Language Models (LLMs) integrated with external tools. Adversarial images, visually indistinguishable from benign images, can manipulate the LLM to execute unintended tool commands, compromising the confidentiality and integrity of user resources. The attack is effective across diverse prompts, remaining stealthy both in the image itself and in the generated text response.

Examples: See arXiv:2405.18540. The paper provides specific examples of adversarial images causing the LLM to delete emails, send emails, and book travel using integrated tools. These examples demonstrate the generalizability of the attack to different tool invocation syntaxes and various prompts.

Impact: Successful exploitation allows attackers to control the LLM's interaction with connected tools, leading to data breaches (confidentiality), data modification or deletion (integrity), and potentially financial loss (availability). The stealthy nature enhances the likelihood of successful attacks going unnoticed by users.

Affected Systems: Multimodal LLMs which accept images as input and are integrated with external tools using a function-call or similar invocation mechanism, including, but not limited to, systems employing LangChain or Semantic Kernel frameworks. Specific systems affected will depend on the underlying LLM and tool integrations. The paper demonstrates the vulnerability on LLaMA Adapter, but attacks may be transferable to other systems.

Mitigation Steps:

• Input Sanitization: Implement robust image sanitization techniques to detect and reject adversarial images. This could involve methods that are resistant to adversarial attacks or incorporating checksums for images to verify their integrity.

• Tool Access Control: Enforce strict access control for LLMs interacting with tools. Restrict the LLM's ability to execute sensitive operations unless explicitly authorized by the user. Granular permissions should be implemented to minimize the impact of successful attacks.

• Output Verification: Verify the LLM's tool invocation commands before execution. This involves comparing generated commands against expected patterns or user-provided constraints.

• Model Defenses: Develop LLM architectures and training methods more resistant to adversarial attacks against image inputs. This is a more long-term solution requiring substantial research and development.

• Security Monitoring: Implement monitoring mechanisms to detect anomalous tool invocations. This could include auditing logs of tool interactions and establishing baselines to detect deviations.