Multimodal Linking Visual Insecurity

Description: Multimodal Entity Linking (MEL) systems, encompassing both traditional dual-encoder models and Multimodal Large Language Models (MLLMs), are vulnerable to gradient-based white-box adversarial attacks. By applying imperceptible perturbations to visual inputs via Projected Gradient Descent (PGD), Auto-PGD (APGD), or Carlini & Wagner (CW) methods, an attacker can manipulate the visual embeddings generated by the model. This manipulation disrupts the cross-modal alignment structure, causing the model to incorrectly link visual content to unrelated entities in a knowledge base during Image-to-Text (I2T) and Image+Text-to-Text (IT2T) tasks. The vulnerability stems from the models' reliance on visual inputs that lack sufficient robustness against noise when textual context is absent or insufficient.

Examples: The vulnerability is reproducible using the specific adversarial generation parameters and dataset provided by the researchers.

• Repository: See https://anonymous.4open.science/r/MEL-Robustness-90A5 for the MEL adversarial example dataset constructed on Wikidata-MEL, Richpedia-MEL, WikiDiverse, WIKIPerson, and M3EL.
• Attack Vector Implementation: To reproduce the attack, the adversary calculates the gradient of the loss function with respect to the input image pixels:

1. PGD/APGD: Maximize the Cross-Entropy loss between the model logits and the ground-truth label, constrained by an $L_{\infty}$ ball. $$ \max_{|\delta|_{\infty}\leq\epsilon}\mathcal{L}(f(x+\delta),y) $$ (Where $\epsilon=8/255$ for Normal attacks and $\epsilon=0.2$ for Strong attacks).
2. CW Attack: Minimize the sum of the $L_2$ distance of the perturbation $\delta$ and the objective function $f$.
3. Result: In the case of Qwen2.5-VL on the WIKIPerson dataset, a CW-Strong attack results in a 35.8% drop in linking accuracy.

Impact:

• Entity Misidentification: Successful attacks cause the system to link visual subjects to incorrect knowledge graph entities (e.g., misidentifying a specific person or object).
• Downstream Corruption: Compromises systems relying on MEL for Knowledge-Enhanced Question & Answering (QA), Image-Text Retrieval, and Open-Domain Entity Alignment.
• Model Instability: High susceptibility to visual noise leads to unpredictable behavior in real-world deployments where image quality varies.

Affected Systems:

• Traditional MEL Models: ALIGN, BLIP, CLIP, FLAVA, OWL-ViT, SigLIP.
• Multimodal Large Language Models (MLLMs): LLaVA (based on LLaMA/Vicuna), Qwen2.5-VL, MiniGPT-4.

Mitigation Steps:

• Implement LLM-RetLink (LLM and Retrieval-Augmented Entity Linking): Adopt a two-stage architecture that integrates Large Vision Models (LVMs) with web-based dynamic retrieval.
• Extract Explicit Descriptors: Use LVMs to automatically extract word-level entity descriptors (e.g., "apple", "fruit") from input images to serve as query cues, reducing reliance on raw visual embeddings.
• Retrieval-Augmented Context: Dynamically query external knowledge bases (e.g., Wikidata, Wikipedia) to obtain sentence-level descriptions associated with candidates.
• LLM-Based Disambiguation: Feed the retrieved content into an LLM to perform semantic matching between candidates and context, rather than relying on static entity descriptions or pure visual alignment.
• Contextual Integration: Ensure textual information is incorporated alongside visual input, as experimental results indicate that contextual semantic information partially mitigates the impact of visual perturbations.