Skip to main content
LLM Security Database
Skip to research search
Last analyzed 9/9/2026

Language Model Security Database

985 research findings · 1123 evaluated models

Filtered research findings

56 entries

Matches every word across titles, descriptions, sources, affected systems, and models.

Deleting a memory record can leave its information in an agent's summaries, pending plans and KV cache. The paper evaluates revocation across this derived execution state.

Forgetting Without Restarting: Execution-State Unlearning for Stateful LLM Agents
Evaluated models: Llama 3.1 8B Instruct, Qwen 2.5 7B Instruct, Mistral 7B Instruct v0.3

Source: arXiv

Published 9/2/2026
Analyzed 9/9/2026

Document-image PII detection can miss identifiers despite improving average localization scores. LeakageBench evaluates 500 pages with 11,954 annotations.

LeakageBench: Document-Level Leakage Risk for Redacting Personally Identifiable Information in Document Images
Evaluated models: GLiNER-base, GLiNER-multi-PII, NVIDIA GLiNER PII +5 more

Source: arXiv

Published 8/26/2026
Analyzed 9/9/2026

A compromised serving framework can violate user-data isolation through shared GPU state. GIFT evaluates per-user information-flow enforcement in vLLM and DistServe.

Here is a GIFT: Enforcing User Data Isolation in LLM Serving via GPU Information Flow Tracking
Evaluated models: Qwen 2.5 14B, Qwen 2.5 32B, Qwen 2.5 72B +3 more

Source: arXiv

Published 7/29/2026
Analyzed 8/13/2026

MemSecBench follows malicious agent-memory content from initial write through persistence, retrieval, action selection, execution, and attempted selective repair. Its controlled Write–Execute–Forget protocol evaluates 310 human-reviewed cases across two harnesses, four memory backends, three model backends, and seven evidence-gated lifecycle checkpoints.

MemSecBench: Tracking Agent Memory Poisoning from Persistence to Consequence and Repair
Evaluated models: GPT-5.5, DeepSeek V4-Pro, MiniMax-M3

Source: arXiv

AgentS4D measures unsafe actions and state changes across complete workspace-agent executions rather than treating task completion or isolated model responses as safety evidence. Its 328 sandboxed cases introduce risky content through user requests, documents, web resources, tools, third-party skills, and persistent memory, then compare the same cases across four agent harnesses and five model backends.

AgentS4D: Benchmarking Runtime Risks across the Execution Lifecycle of LLM-Based Workspace Agents
Evaluated models: GPT-5.5, Gemini 3.1 Pro, DeepSeek V4-Pro +2 more

Source: arXiv

Published 7/28/2026
Analyzed 8/13/2026

The MTGuard study evaluates unsafe Model Context Protocol tool calls originating from compromised server data, host-side execution changes, and malicious user-controlled resources. Its hybrid monitor combines pre-execution parameter inspection, behavioral observation, and post-execution result verification across browser-automation and financial-analysis agents.

Hybrid Analysis for Secure MCP Tool Use in LLM Agents
Evaluated models: GPT-5.6 Luna, DeepSeek V4 Flash, DeepSeek V4-Pro

Source: arXiv

Published 7/28/2026
Analyzed 8/13/2026

HANDBOOK.md measures whether an agent can apply detailed organizational rules while completing realistic, multi-step enterprise tasks. The vendor-authored benchmark includes 65 resettable MCP-backed workflows, policy documents of 20 to 124 pages, and 824 deterministic rubric checks covering required decisions, prohibited actions, and final environment state.

HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following
Evaluated models: Claude Fable 5, GPT-5.6 Sol, Claude Opus 4.8 +16 more

Source: arXiv

Published 7/22/2026
Analyzed 8/13/2026

OpenSkillRisk evaluates whether agent harnesses safely handle third-party skills that introduce risky behavior through otherwise plausible, benign tasks. The benchmark assembles 263 risky skills from public agent-skill ecosystems and tests three CLI-agent harnesses against seven risk categories using isolated task workspaces, mocked external services, and execution-level evidence.

OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills
Evaluated models: GPT-5.1 Codex Mini, GPT-5.3 Codex, GPT-5.4 +10 more

Source: arXiv

Research methodology

Entries summarize publicly available primary-source security research. Model names reflect only systems explicitly evaluated by the cited paper, and measurements are research-reported unless independent verification is stated.