Attack LLMs with Toxic Answers
Large Language Models (LLMs) are vulnerable to a targeted jailbreak attack, termed Atoxia, which can force the generation of specific harmful content. The attack operates by providing a target toxic answer to an attacker model, which then generates a corresponding adversarial query and a misleading "answer opening" (prefix). When the query and the answer prefix are presented to a vulnerable LLM, the model is induced to continue the generation, bypassing its safety alignment and completing the…
Evaluated models: GPT-3.5 Turbo, GPT-4, GPT-4o+5 more
- GPT-3.5 Turbo
- GPT-4
- GPT-4o
- GPT-4o Mini
- Llama 2 7B Chat
- Llama 3 8B Instruct
- Mistral 7B
- Vicuna 7B v1.5