Skip to main content
LLM Security Database
Skip to research details
Back to research findings

PII leakage in document-image redaction benchmarks

Document-image PII detection can miss identifiers despite improving average localization scores. LeakageBench evaluates 500 pages with 11,954 annotations.

Published
Analyzed
Paper-reported evidence
Primary source linked
Read primary paper
Cite & share
Source BibTeX

Citation metadata is maintained by the primary source and may reflect a later revision.

Paper-evaluated models(8)

GLiNER-base, GLiNER-multi-PII, NVIDIA GLiNER PII +5 more
  • GLiNER-base
  • GLiNER-multi-PII
  • NVIDIA GLiNER PII
  • GLiNER2
  • Qwen3-VL-32B
  • InternVL3 38B
  • GPT-5.4
  • GPT-5.5
On this page

Description

Document-image PII detection can miss identifiers despite improving average localization scores. LeakageBench evaluates 500 pages with 11,954 annotations.

Examples

See the primary benchmark (opens in a new tab).

Impact

GPT-5.5 with Code Interpreter raises localization F1 from 0.090 to 0.249, yet critical page leakage remains 0.968 under typed box matching at IoU 0.75. This is a detection proxy on a challenge set, not an audit of released redacted files.

Affected Systems

  • Evaluated OCR, PII-detector and vision-language document workflows, including Code Interpreter-assisted inspection.

Mitigation Steps

  • Measure missed identifiers per page alongside aggregate F1.
  • Include linkage identifiers and verify OCR/box alignment.
  • Inspect the final rendered redaction before release.

Evidence

Research context and provenance

Research topics
Application layer
Vision
Multimodal
Data privacy
Data security
Reliability
Catalog identifier
LMVD-1b9be65b
Internal research identifier, not an official CVE identifier.
Evidence and verification
Paper-reported; independent reproduction is not documented.
Primary source plus a dedicated evidence section.
Severity
Not rated by this catalog.
Source and publication type
arXiv · Research preprint.
Peer-review status is not provided by this source.
Author and publication status
Author metadata is not stored; see the primary paper.
Threat model and attacker access
Not explicitly classified; consult the primary paper.
Related deployment categories
Vision-language models
Taxonomy labels only; paper-specific deployment prerequisites are not inferred.
Affected systems
Evaluated OCR, PII-detector and vision-language document workflows, including Code Interpreter-assisted inspection.

Research Paper

LeakageBench: Document-Level Leakage Risk for Redacting Personally Identifiable Information in Document Images

Primary source: arXiv. Findings are reported by the cited research and have not been independently verified.

View Paper

Related research