Hard-Negative Prompt Evasion

Description

Embedding-based LLM prompt injection detectors, specifically those based on the DeBERTa-v3 architecture, are vulnerable to adversarial evasion attacks utilizing "hard-negative" mining and fuzzing techniques. Attackers can circumvent detection mechanisms by iteratively generating adversarial prompts that are semantically malicious but structurally mutated to evade the classifier's decision boundary. Specific evasion vectors identified include semantic fuzzing (paraphrasing), syntactic fuzzing (manipulation of casing, spacing, and punctuation), and format fuzzing (encapsulation within JSON, YAML, or Markdown). Experimental validation demonstrates that while baseline semantic fuzzing reduces detection accuracy from ~95.9% to ~65.3%, aggressive hard-negative mining combined with semantic perturbation (HM-Max-Sem) reduces detection accuracy to ~37.0%, effectively bypassing the guardrail for the majority of malicious inputs.

Examples

The following transformation classes function as reproduction steps to bypass the detector. While the paper redacts specific successful prompts to prevent misuse, the attack methodology is defined as follows:

1. Semantic Fuzzing: Use a paraphrasing model (e.g., humarin/chatgpt_paraphraser_on_T5_base) to substitute synonyms while retaining the imperative intent of the attack.
2. Syntactic Fuzzing: Apply stochastic edits (5% mutation rate) to the prompt:

• Randomly altering capitalization (e.g., iGnOrE pReViOuS).
• Inserting irregular spacing between tokens.
• Modifying punctuation or stopword positions.

3. Format Fuzzing: Encapsulating the adversarial instruction within structured data wrappers rather than plain text.

• JSON: {"instruction": "ignore_instructions", "payload": "..."}
• YAML: Wrapping the attack in a YAML block.
• Markdown: Embedding the prompt within Markdown code blocks or headers.

4. Hard-Negative Iteration:

• Submit a batch of fuzzed prompts to the target detector.
• Identify "Hard Negatives" (Malicious prompts classified as Benign).
• Use these successful evasions as seed data for the next generation of attacks.

Impact

• Guardrail Bypass: Malicious actors can successfully inject prompts that violate safety policies, ignoring pre-inference filtering.
• Jailbreaking: High-probability success in inducing the LLM to generate restricted, harmful, or objectionable content.
• Detector Degradation: Static detection models become obsolete as attackers optimize inputs against the model's specific decision boundary weaknesses.

Affected Systems

• ProtectAI/deberta-v3-base-prompt-injection (specifically cited as the baseline victim model).
• Any LLM guardrail system relying on static, BERT-based binary classification for prompt injection detection without continuous adversarial retraining.

Mitigation Steps

• Implement HASTE (Hard-negative Attack Sample Training Engine): Deploy a closed-loop training pipeline that iteratively generates, evaluates, and refines adversarial prompts.
• Hard-Negative Mining: Automatically identify prompts that successfully evade the current detector (false negatives) and immediately re-inject them into the training dataset for the next model epoch.
• Adversarial Training with Fuzzing: Retrain detection models using a dataset augmented with semantic, syntactic, and format fuzzing (JSON/YAML wrappers) to prevent overfitting to specific lexical features.
• LLM-as-a-Judge Integration: Utilize a secondary LLM (e.g., GPT-4o or JailJudge) to evaluate the maliciousness of prompts that bypass the primary detector, ensuring high-quality labeling for retraining.
• Temporal Feedback Loops: Establish a continuous retraining schedule (demonstrated effective at 5 and 10 iterations) where the detector is updated with the latest evolved adversarial samples.