Role-Confusion Prompt Injection via Forged Reasoning

Description

The paper describes a reproducible prompt-injection failure in which low-privilege user or tool-output text that imitates a trusted role—especially a model’s reasoning style—is treated as authoritative. Its zero-shot, black-box “CoT Forgery” evaluation injects fabricated reasoning into user prompts and retrieved webpage/tool output. The authors report 60% average attack success on StrongREJECT and 56–70% success in an agent data-exfiltration evaluation, versus near-zero or substantially lower baselines. These are paper-reported measurements, not independently verified facts.

Examples

See the primary source for the paper's evaluation and defensive reproduction context: https://arxiv.org/abs/2603.12277 (opens in a new tab)

Impact

Successful role confusion can bypass safety refusals or cause tool-using agents to treat untrusted webpage content as user authorization, leading to unsafe outputs, unauthorized tool actions, and exfiltration of sensitive workspace data. The paper’s agent scenario used simulated credentials and measured attempted or completed exfiltration.

Affected Systems

• Reasoning-enabled chat assistants that accept attacker-controlled user text
• Tool-using or browsing agents that ingest untrusted webpages or tool outputs
• Agent runtimes with shell, filesystem, network, or other consequential tools
• Applications relying primarily on role tags or prompt delimiters as an instruction/data security boundary

Mitigation Steps

• Treat all retrieved content and tool output as untrusted data; never allow it to authorize tool calls or override the original user task.
• Enforce least privilege and explicit user approval outside the model for sensitive file access, credential handling, network egress, uploads, and other consequential actions.
• Use deterministic policy checks and allowlists for tool arguments and destinations; do not rely solely on model refusal behavior or role tags.
• Sanitize or structurally isolate untrusted content and flag role-like declarations, chat-template markers, and reasoning-style text before it reaches a privileged agent context.
• Add defensive regression tests using synthetic secrets and non-networked endpoints; the authors provide role-probe and reproduction notebooks, but operational attack payloads should not be used in production environments.
• Pursue model-level defenses that make architectural role/source signals robust in representations, such as role-salience marking, dedicated embeddings, or role-separation training; the paper discusses these as defense directions, not validated fixes.

Evidence

• Primary source submission metadata: https://arxiv.org/abs/2603.12277 (opens in a new tab) — records the paper date as 2026-02-22, which is used for this entry.
• Primary source abstract and Introduction: https://arxiv.org/html/2603.12277 (opens in a new tab) — Abstract and §1 define role confusion and report that CoT Forgery injects fabricated reasoning into user prompts and tool outputs, with 60% attack success across frontier models.
• Primary source §3.1–§3.2: https://arxiv.org/html/2603.12277 (opens in a new tab) — the authors describe a strict zero-shot, black-box attack evaluated on 313 StrongREJECT requests against gpt-oss-20b, gpt-oss-120b, o4-mini, and GPT-5 nano/mini/full; §3.2 reports 0–4% baseline ASR and 17–52% for GPT-5 variants, with gpt-oss and o4-mini exceeding 80%.
• Primary source §3.3: https://arxiv.org/html/2603.12277 (opens in a new tab) — the agent evaluation embeds injections in webpage tool output for a ReAct-style agent with shell access and a simulated .env file; the paper reports standard-injection ASR of 0–2% except gpt-oss-20b at 26%, while CoT Forgery reaches 56–70%.
• Primary source §3.4: https://arxiv.org/html/2603.12277 (opens in a new tab) — the style ablation preserves semantics while removing reasoning-style markers and reports ASR falling from 61% to 10%, supporting the paper’s causal role-confusion claim.
• Primary source §4.2: https://arxiv.org/html/2603.12277 (opens in a new tab) — under conflicting tags, CoT-style text wrapped as user input reportedly retains 85% CoTness and 2% Userness; cross-model tests report user-style text wrapped as tool output retaining 76–88% Userness.
• Primary source §5.1–§5.2: https://arxiv.org/html/2603.12277 (opens in a new tab) — the authors report attack success rising from 9% to 90% across CoTness quantiles and from 2% to 70% across Userness quantiles in 1,000 agent attempts; these are author-reported measurements.
• Primary source §6, “Defenses Against Prompt Injection”: https://arxiv.org/html/2603.12277 (opens in a new tab) — discusses detection/sanitization, role-salience marking, embeddings, and training as defense directions, while concluding defenses must address attacker-controllable role-perception features.
• Primary source Appendix A and B: https://arxiv.org/html/2603.12277 (opens in a new tab) — states that replication code and standalone role-probe notebooks are available and documents model selection, generation, evaluation, and the simulated agent environment.
• Directly linked artifact: https://github.com/role-confusion/prompt-injection-as-role-confusion (opens in a new tab) — repository README lists role-probe demos, CoT Forgery evaluations, agent prompt-injection notebooks, and full paper-reproduction workflows.
• Primary source Appendix B.3: https://arxiv.org/html/2603.12277 (opens in a new tab) — reports additional cross-family CoT Forgery results of 92% for GLM-4.6, 84% for Kimi-K2-Instruct, and 74% for Minimax-M2; these figures were not independently verified.