Skip to main content
LLM Security Database
Skip to research details
Back to research findings

LLM Judge Prompt Injection

Published
Analyzed
Paper-reported evidence
Primary source linked
Read primary paper
Cite & share
Source BibTeX

Citation metadata is maintained by the primary source and may reflect a later revision.

Paper-evaluated models(2)

  • Falcon 3 3B Instruct
  • Qwen 2.5 3B Instruct
On this page

Description

Large Language Models (LLMs) used for evaluating text quality (LLM-as-a-Judge architectures) are vulnerable to prompt-injection attacks. Maliciously crafted suffixes appended to input text can manipulate the LLM's judgment, causing it to incorrectly favor a predetermined response even if another response is objectively superior. Two attack vectors are identified: Comparative Undermining Attack (CUA), directly targeting the final decision, and Justification Manipulation Attack (JMA), altering the model's generated reasoning.

Examples

See arXiv:2505.13348 (opens in a new tab) for specific examples of adversarial suffixes generated using the Greedy Coordinate Gradient (GCG) optimization method. Examples show successful attacks resulting in over 30% Attack Success Rate (ASR) for CUA and 15-17% for JMA.

Impact

Compromised evaluation leading to inaccurate assessments of generated text quality. This could impact multiple applications that rely on LLMs for evaluation such as Reinforcement Learning from Human Feedback (RLHF), automated validation in crowdsourcing, and intelligent search and retrieval systems. Attackers can manipulate the LLM's judgment to promote inferior content or suppress superior content.

Affected Systems

Systems employing open-source instruction-tuned LLMs (such as Qwen2.5-3B-Instruct and Falcon3-3B-Instruct) in LLM-as-a-Judge architectures, or similar models vulnerable to prompt injection.

Mitigation Steps

  • Develop and implement robust defense mechanisms against prompt injection attacks. The paper suggests exploring techniques like goal prioritization and attention tracking.
  • Utilize adversarial training techniques to improve the robustness of LLM-as-a-Judge models to prompt injection.
  • Employ input sanitization and filtering techniques to identify and remove potentially malicious suffixes.
  • Regularly audit and update LLM models to address vulnerabilities as they are discovered.
  • Implement verification methods to independently validate the evaluations produced by the LLM-as-a-Judge system.

Research context and provenance

Research topics
Prompt layer
Injection
Application layer
Black-box
Integrity
Safety
Catalog identifier
LMVD-886657fa
Internal research identifier, not an official CVE identifier.
Evidence and verification
Paper-reported; independent reproduction is not documented.
Primary research source linked.
Severity
Not rated by this catalog.
Source and publication type
arXiv · Research preprint.
Peer-review status is not provided by this source.
Author and publication status
Author metadata is not stored; see the primary paper.
Threat model and attacker access
Black-box model, service, or application access.
Related deployment categories
No related deployment category is classified.
Taxonomy labels only; paper-specific deployment prerequisites are not inferred.
Affected systems
Systems employing open-source instruction-tuned LLMs (such as Qwen2.5-3B-Instruct and Falcon3-3B-Instruct) in LLM-as-a-Judge architectures, or similar models vulnerable to prompt injection.

Research Paper

Investigating the Vulnerability of LLM-as-a-Judge Architectures to Prompt-Injection Attacks

Primary source: arXiv. Findings are reported by the cited research and have not been independently verified.

View Paper

Evidence

This entry is based on a primary research source. Its findings are paper-reported; independent reproduction and verification are not claimed.

https://arxiv.org/abs/2505.13348

Related research