PolyJailbreak Cross-Modal Safety Asymmetry

Description

The paper describes a reproducible black-box evaluation and attack framework, PolyJailbreak, for multimodal LLMs. It reports that uneven text-versus-vision safety alignment allows jointly optimized text and image inputs to bypass refusal behavior without model internals. The authors attribute this to visual alignment weakening textual refusal representations and to cross-modal fusion making harmful intent harder to separate from benign intent. These are paper-reported findings, not independently verified facts.

Examples

See the primary source for the paper's evaluation and defensive reproduction context: https://arxiv.org/abs/2510.17277 (opens in a new tab)

Impact

Successful exploitation can cause safety-aligned multimodal models to produce policy-violating responses across criminal, harassment, hate, misinformation, self-harm, terrorism, violence, and weapons categories. The paper reports an average 83.34% attack success rate and 3.976/5 harmfulness score across eight models; reported ASR exceeded 95% for GPT-4o, Gemini-2.5-Flash, LLaVA-1.5, and LLaVA-1.6. The authors note classifier-based judging and model-specific variance as limitations.

Affected Systems

• Safety-aligned multimodal large language models accepting combined text and image inputs
• MLLM deployments whose text and vision safety controls are evaluated separately rather than jointly
• Models using trainable visual alignment that may alter backbone refusal behavior

Mitigation Steps

• Add cross-modal red-team evaluations using benign surrogate tasks that test text-image semantic conflicts, visual obfuscation, and coordinated modality changes without harmful payloads.
• Apply safety classification and policy reasoning to the joint meaning of text and image inputs, not only to each modality independently.
• Preserve and regression-test text-only refusal behavior before and after visual alignment or multimodal fine-tuning.
• Use adaptive multimodal screening and semantic safety constraints; the paper reports AdaShield and ECSO reduced ASR more than simple text perturbation defenses, though neither fully suppressed attacks.
• Monitor repeated black-box probing, model-specific refusal-template discovery, and high-variation multimodal requests; rate-limit or escalate suspicious sessions.
• Continuously evaluate defenses against diverse and long-tail strategy combinations rather than relying on fixed keyword, template, or image-only filters.

Evidence

• Primary source abstract and Sections III-C, V, and VI: https://arxiv.org/html/2510.17277 (opens in a new tab) — defines a black-box adversary limited to text, image, or combined input-output queries and describes PolyJailbreak's composable cross-modal optimization framework.
• Primary source Section III-C: https://arxiv.org/html/2510.17277 (opens in a new tab) — specifies single-turn black-box access, no parameter, architecture, or training-data access, making the evaluation security-relevant and reproducible at the methodology level.
• Primary source Section IV-A: https://arxiv.org/html/2510.17277 (opens in a new tab) — describes a 400-pair dataset across eight sensitive categories and defines attack success using an external safety classifier.
• Primary source Section VI-A: https://arxiv.org/html/2510.17277 (opens in a new tab) — lists the eight evaluated open- and closed-source target models, metrics, official API testing for commercial systems, and evaluation components.
• Primary source Table VI and Section VI-B: https://arxiv.org/html/2510.17277 (opens in a new tab) — reports PolyJailbreak ASR/HS results, including 83.34% average ASR, 3.976 average HS, and over 95% ASR on four models; these measurements are reported by the authors and were not independently verified.
• Primary source Table VII and Section VI-C: https://arxiv.org/html/2510.17277 (opens in a new tab) — reports that jointly optimized text-image inputs achieved higher ASR than original instructions, optimized text alone, or optimized images alone across all eight models.
• Primary source Section VI-D and Table IX: https://arxiv.org/html/2510.17277 (opens in a new tab) — reports reduced but persistent attack success under SmoothLLM, AdaShield, and ECSO defenses.
• Primary source Sections VII-VIII: https://arxiv.org/html/2510.17277 (opens in a new tab) — states limitations, withholds harmful outputs and step-by-step instructions, and recommends multimodal jailbreak testing and defenses that jointly reason over visual and textual inputs.
• Primary source metadata: https://arxiv.org/abs/2510.17277 (opens in a new tab) — records initial submission on 2025-10-20 and current title/version information.