Agent Action Hijacking

CVE-2024-XXXX

**Description:** A vulnerability in LLM-based agents, dubbed AI Agent Injection (AI²), allows attackers to hijack the agent's actions by manipulating the agent's memory retrieval mechanism. The attack involves two main steps: (1) Stealing action-aware knowledge from the agent's memory using crafted adversarial queries targeting the retriever module and (2) Generating Trojan prompts consisting of a Trojan string and hijacking instructions. The Trojan string is designed to manipulate the retriever into retrieving specific knowledge related to the target malicious action, while bypassing safety filters. The hijacking instructions then use this retrieved knowledge, assembled with parts of the original benign user's input, to construct harmful instructions. The use of harmless prompts that leverage knowledge theft makes this attack stealthy and effective against black-box agent systems.

**Examples:**

See [arXiv:2406.10864](https://arxiv.org/abs/2406.10864) for specific examples of AI² attacks.
Example 1 (Text-to-SQL Agent):
*   Target Action: Delete user data from SQL agent.
*   Knowledge Stealing: The attacker first crafts natural language queries designed to retrieve knowledge related to the "DELETE" operation. These queries are designed to extract the relevant SQL command descriptions from the agent's memory.
*   Trojan Prompt:  The attacker then crafts a Trojan prompt, such as "Perform the same operation on the Artist table."  The Trojan string portion of this prompt is crafted to trick the retriever into selecting knowledge about 'DELETE' operations. The "Hijacking Instructions" section of the prompt is used to instruct the agents to use the malicious action "delete" based on the stolen knowledge.
*   Result: The agent, using its reasoning module, combines the Trojan prompt with the retrieved knowledge to generate and execute a SQL DELETE command, such as "DELETE FROM users WHERE id = X". This circumvents safety measures as no direct "DELETE" command is invoked.

**Impact:**
*   Unauthorized data access and modification.
*   Data deletion, leading to denial of service.
*   Execution of arbitrary code/commands.
*   Compromise of agent's integrity and trust.

**Affected Systems:**
*   LLM-based agents that utilize a memory component (e.g., long-term memory or knowledge bases) for storing and retrieving information.
*   Agents that use a retriever module to fetch relevant information based on user queries.
*   Agents employing safety filters (e.g., banned word filters, forbidden operation filters) that are designed to mitigate prompt-injection and jailbreak attacks.
*  Text-to-SQL agents, open-domain Question and Answer(Q & A) agents and other agent-based environments.

**Mitigation Steps:**

*   **Enhance Retriever Security:** This includes developing robust methods to prevent unauthorized access to the knowledge base through memory poisoning and knowledge leakage.
*   **Safety-Enhanced Training**: Train the LLM's safety capabilities in the decision process, to enforce secure behavior regarding different kinds of operations.
*   **Combined Prompt Analysis:** Implement a component to check all the components of the prompts to protect the agent on the final-stage.
*   **Enhanced Tool Security:** Ensure the tools do not execute actions outside of the agent's planned operation so as to reduce the impact in attacks.
*   **User Behavior Monitoring:** Implement a monitoring system to detect any anomaly behavior of the agents.