Skip to main content
LLM Security Database
Skip to research details
Back to research findings

Selective refusal gaps in visual question answering

KoNA measures whether vision-language models answer valid image questions while refusing unsafe components or correcting unsupported premises. Its 9,300 question-answer pairs include mixed and fully answerable controls.

Published
Analyzed
Paper-reported evidence
Primary source linked
Read primary paper
Cite & share
Source BibTeX

Citation metadata is maintained by the primary source and may reflect a later revision.

Paper-evaluated models(8)

InternVL3 2B Instruct, InternVL3-78B-Instruct, Qwen 2.5 VL 3B Instruct +5 more
  • InternVL3 2B Instruct
  • InternVL3-78B-Instruct
  • Qwen 2.5 VL 3B Instruct
  • Qwen 2.5 VL 72B Instruct
  • GPT-5
  • Gemini 2.5 Flash
  • InternVL3-2B-KoNA
  • Qwen2.5-VL-3B-KoNA
On this page

Description

KoNA measures whether vision-language models answer valid image questions while refusing unsafe components or correcting unsupported premises. Its 9,300 question-answer pairs include mixed and fully answerable controls.

Examples

See the primary benchmark (opens in a new tab).

Impact

Default compound-query Safety accuracy is 0.01 for Qwen2.5-VL-3B and 0.91 for GPT-5. KoNA tuning raises the small Qwen model to 0.98, while fully answerable accuracy moves from 0.73 to 0.71. Tuning covers two small open models; this is a selective-response benchmark, not adaptive jailbreak testing or a production failure estimate.

Affected Systems

  • The six evaluated vision-language models and two KoNA-tuned variants; GPT-5-mini serves as an evaluator.

Mitigation Steps

  • Score valid and unsafe components separately.
  • Evaluate answerability alongside refusal behavior after tuning.
  • Use human sampling to audit automatic safety and factuality judgments.

Evidence

Research context and provenance

Research topics
Model layer
Vision
Multimodal
Safety
Reliability
Hallucination
Fine-tuning
Catalog identifier
LMVD-41ff4d29
Internal research identifier, not an official CVE identifier.
Evidence and verification
Paper-reported; independent reproduction is not documented.
Primary source plus a dedicated evidence section.
Severity
Not rated by this catalog.
Source and publication type
arXiv · Research preprint.
Peer-review status is not provided by this source.
Author and publication status
Author metadata is not stored; see the primary paper.
Threat model and attacker access
Not explicitly classified; consult the primary paper.
Related deployment categories
Vision-language models; Fine-tuning
Taxonomy labels only; paper-specific deployment prerequisites are not inferred.
Affected systems
The six evaluated vision-language models and two KoNA-tuned variants; GPT-5-mini serves as an evaluator.

Research Paper

Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models

Primary source: arXiv. Findings are reported by the cited research and have not been independently verified.

View Paper

Related research