Multilingual Flowchart Jailbreaks in Vision-Language Models

Description

MLingualFC is a reproducible black-box safety evaluation showing that harmful instructions rendered as multilingual flowchart images can bypass vision-language model safeguards more often than equivalent text-only inputs. The paper evaluates horizontal, vertical, and tortuous layouts across English, Hindi, Punjabi, Spanish, Romanian, and German. Reported results vary substantially by language, script, layout, and model; these are paper-reported measurements, not independently verified findings. The authors provide evaluation code but restrict the full harmful dataset, so defensive reproduction should use benign policy-violating surrogates or authorized red-team datasets and compare refusal behavior across equivalent text and image-rendered prompts.

Examples

See the primary source for the paper's evaluation and defensive reproduction context: https://arxiv.org/abs/2606.07706 (opens in a new tab)

Impact

Successful exploitation can cause multimodal assistants to produce unsafe or policy-violating responses when harmful intent is encoded visually and in non-English languages. English-only or text-only safety testing may underestimate risk. Layout-dependent behavior also creates reliability gaps: the same semantic content can receive materially different safety treatment depending on language, script, and flowchart structure.

Affected Systems

• Multilingual vision-language models that accept image-plus-text prompts
• Safety filters or alignment mechanisms evaluated primarily on English or text-only inputs
• Applications exposing black-box multimodal inference interfaces

Mitigation Steps

• Add multilingual, multimodal red-team tests covering equivalent text, text-as-image, and structured-flowchart inputs across scripts and layouts.
• Apply safety classification to OCR-extracted image text and visual structure before generation, with policy checks shared across text and image channels.
• Train and evaluate refusal behavior on semantically equivalent multilingual visual prompts, including Latin and non-Latin scripts.
• Measure safety separately from visual-language comprehension so low attack success caused by OCR or script-recognition failures is not mistaken for robust alignment.
• Use layout-invariant safety evaluation and regression tests for horizontal, vertical, and complex visual arrangements.
• Monitor production multimodal requests for cross-lingual and image-encoded policy-evasion patterns while preserving legitimate multilingual use.

Evidence

• arXiv abstract and Section 3.1 report a black-box, single-turn threat model using multilingual structured flowchart images to bypass MVLM safety safeguards: https://arxiv.org/html/2606.07706#S3.SS1 (opens in a new tab)
• Section 3.2 describes the benchmark construction: translated flowchart images in Hindi, Punjabi, Spanish, Romanian, and German with horizontal, vertical, and tortuous layouts, paired with benign textual instructions: https://arxiv.org/html/2606.07706#S3.SS2 (opens in a new tab)
• Section 4 identifies the evaluated targets as Qwen2.5-VL-3B-Instruct, Gemma-4-E4B-it, and Pangea-7B, and Section 3.3 describes GPT-oss-120b judging with scores of at least 4 counted as harmful: https://arxiv.org/html/2606.07706#S4 (opens in a new tab) and https://arxiv.org/html/2606.07706#S3.SS3 (opens in a new tab)
• Table 1 reports flowchart-plus-harmless-text ASR values including Qwen2.5-VL at 89.36% for Spanish tortuous and German tortuous layouts, Gemma-4 at 74.47% for Hindi horizontal, and Pangea at 72.00% for English vertical; these are author-reported measurements: https://arxiv.org/html/2606.07706#S4 (opens in a new tab)
• Section 5.1 reports that attack success varies by language and layout, with Qwen2.5-VL most vulnerable across Spanish, Romanian, and German and Gemma-4 particularly vulnerable in Hindi; it also states English-only evaluation can underestimate multilingual vulnerabilities: https://arxiv.org/html/2606.07706#S5.SS1 (opens in a new tab)
• Section 5.1.1 compares flowcharts against text baselines and reports substantial ASR drops for most languages without visual flowcharts, supporting the security relevance of the visual encoding: https://arxiv.org/html/2606.07706#S5.SS1.SSS1 (opens in a new tab)
• Section 5.2 and the conclusion caution that low Hindi and Punjabi ASR can reflect weak multilingual visual understanding rather than stronger safety alignment: https://arxiv.org/html/2606.07706#S5.SS2 (opens in a new tab) and https://arxiv.org/html/2606.07706#S6 (opens in a new tab)
• The directly linked repository documents translation, plain-text baseline, and red-teaming evaluation scripts, while the paper's Ethical Statement says the full dataset is not publicly released and is restricted to certified researchers: https://github.com/Rishabhpm23/MLingualFC (opens in a new tab) and https://arxiv.org/html/2606.07706#S6 (opens in a new tab)