Multi-Agent Prompt Permutation Attack

Description

A vulnerability in multi-agent Large Language Model (LLM) systems allows for a permutation-invariant adversarial prompt attack. By strategically partitioning adversarial prompts and routing them through a network topology, an attacker can bypass distributed safety mechanisms, even those with token bandwidth limitations and asynchronous message delivery. The attack optimizes prompt propagation as a maximum-flow minimum-cost problem, maximizing success while minimizing detection.

Examples

See arXiv:2504.00218 (opens in a new tab) for specific examples of attack vectors against Llama, Mistral, Gemma, and DeepSeek LLMs using datasets including JailBreakBench and AdversarialBench.

Impact

Successful exploitation leads to jailbreaks, allowing the attacker to circumvent safety protocols and elicit harmful or unintended outputs from the target LLM agent within the multi-agent system. Impact is dependent on the function of the targeted LLM agent within the broader system but could range from information leakage to system compromise.

Affected Systems

Multi-agent LLM systems utilizing interconnected agents that communicate via a network topology with inherent constraints like limited token bandwidth, latency, and distributed safety mechanisms. Specific models shown to be affected include Llama, Mistral, Gemma, and DeepSeek variants.

Mitigation Steps

• Implement multi-agent specific safety mechanisms that are robust against distributed attacks.
• Design systems to minimize the network's susceptibility to maximum-flow minimum-cost attacks. This may include adjusting network topology or bandwidth constraints.
• Develop more sophisticated safety filters that can detect fragmented adversarial prompts, and account for variations in message arrival order.
• Regularly evaluate and update safety mechanisms using adversarial training techniques to proactively counter emerging attacks.