Universal MLLM Target Matching

Description

Closed-source Multi-modal Large Language Models (MLLMs) are vulnerable to Universal Targeted Transferable Adversarial Attacks (UTTAA). An attacker can generate a single, image-agnostic adversarial perturbation ($\delta$) that, when added to any arbitrary source image, steers the victim model to output a description or classification matching a specific target image chosen by the attacker. This vulnerability exploits the transferability of adversarial features from open-source surrogate vision encoders (e.g., CLIP, ViT) to proprietary models.

The specific attack vector, termed TarVRoM-Attack (Target-View Routed Meta Optimization), overcomes the instability of previous sample-specific attacks through three mechanisms:

1. Target-View Aggregation (TVA) with an Attention-Focused View (AFV): Stabilizes optimization by aggregating gradients over multiple target views and one attention-salient view, rather than a single view.
2. Token Routing (TR): Uses an alignability-gated mechanism to identify source tokens that structurally match the target. It selectively optimizes these tokens for target alignment while constraining non-alignable tokens to preserve original source semantics, preventing optimization drift.
3. Meta-Initialization (MI): Utilizes a Reptile-based meta-learning approach to generate a generalized perturbation initialization ($\delta_0$) that adapts rapidly to new targets with few-shot supervision.

Examples

To reproduce the TarVRoM-Attack against a target MLLM (e.g., GPT-4o) using an ensemble of surrogate encoders $\mathcal{F} = \{f_{\theta_i}\}$:

1. Preparation: Select a target image $\mathbf{x}_{tar}$ and a pool of $N=20$ source images.
2. Meta-Initialization (Stage 1):

• Treat different targets as tasks $\tau$.
• Update a shared initialization $\delta_0$ using the Reptile first-order update rule: $\delta_0 \leftarrow \Pi(\delta_0 + \eta(\bar{\delta} - \delta_0))$, where $\bar{\delta}$ is the average perturbation after inner-loop adaptation on sampled tasks.

3. Target Adaptation (Stage 2):

• Initialize $\delta$ with the learned $\delta_0$.
• For each optimization step, sample a batch of target views $\mathcal{V}^+(\mathbf{x}_{tar})$ including random crops and an Attention-Focused View (centered on peak attention activation).
• Extract features $z_{tar}$ (target) and $z_{as}$ (adversarial source).
• Compute the Token Routing gate $w(z_{as}) = \sigma((r(z_{as}) - \gamma)/\alpha)$ based on cosine similarity $r$ between source and target tokens.
• Optimize $\delta$ using Projected Gradient Descent (PGD) to maximize the objective: $\mathcal{L} = \sum w(z_{as}) \cdot \mathcal{L}_{align}(z_{as}, z_{tar}) + \lambda_{pre} (1-w(z_{as})) \cdot \cos(z_{as}, z_{orig})$

4. Attack Execution:

• Add the resulting universal perturbation $\delta$ to a strictly unseen image $\mathbf{x}_{new}$.
• Input $\mathbf{x}_{new} + \delta$ to the victim model.
• Result: The model generates a caption describing $\mathbf{x}_{tar}$ instead of $\mathbf{x}_{new}$.

Impact

• Targeted Misinformation: Attackers can force MLLMs to identify benign images as harmful content, or harmful images as benign objects, across any user input.
• Scalable Jailbreaking: A single universal perturbation can be distributed to bypass visual safety filters on a mass scale without requiring per-image optimization.
• Service Degradation: The attack generalizes to unseen images, allowing for consistent manipulation of model outputs in real-world API scenarios.

Affected Systems

• GPT-4o (OpenAI)
• Gemini-2.0 (Google)
• Claude Sonnet 4.5 (Anthropic)
• Additional appendix evaluations: GPT-5, GPT-5.2, Gemini 3, and Claude Opus 4.5; Gemini 2.5 is also reported without an exact tier.
• Any MLLM utilizing standard vision-language pre-training alignment (e.g., CLIP, SigLIP) susceptible to transfer attacks.

Mitigation Steps

• Evaluate vision encoders and generated captions against bounded, transferable image perturbations across multiple source images and target concepts.
• Evaluate each input modality and their combination with bounded, inert perturbations; use modality-aware policy checks and reject inputs whose provenance or consistency cannot be established.
• Cross-check safety-sensitive image interpretations against independent perception signals rather than trusting a single vision-language embedding.
• Add a targeted regression using inert data and actions, measure both safety and utility regressions, and monitor production for repeated or adaptive attempts.