Dynamic Security Testing for Third-Party Agent Skills

Description

SkillSentry evaluates third-party agent skills by constructing source-grounded decoy environments and comparing matched executions with and without the tested skill. The method requires completed, observable, skill-attributed side effects rather than treating suspicious text, ordinary privileged operations, or unexecuted paths as proven malicious behavior.

Examples

Use isolated honey environments with synthetic credentials, paired skill-enabled and no-skill control runs, inert external services, and verified before-and-after state; see https://arxiv.org/html/2608.03485 (opens in a new tab).

Impact

The authors report 99.50% recall on a harmful-only benchmark and F1 scores of 96.08% and 96.43% on two mixed benchmarks, with 4.15% and 2.33% false-positive rates respectively. The results come from benchmark packages, generated tasks, simulated decoy resources, and model-based judging; they do not establish complete coverage or production deployment effectiveness.

Affected Systems

• Third-party skill scanners and agent-skill admission workflows
• HarmfulSkillBench, SkillTrustBench, and MalSkillBench evaluation packages
• Hermes- and OpenClaw-configured baseline scanners and skill-enabled MCP or coding-agent environments
• Honey-world execution monitors that track tool calls, sensitive resources, and state changes

Mitigation Steps

• Combine static skill analysis with isolated dynamic execution that exercises source-supported risky behavior.
• Compare skill-enabled runs with matched no-skill controls before attributing an effect to the skill.
• Use synthetic decoy resources and controlled service stubs instead of real credentials or external destinations.
• Require completed, out-of-contract effects and review legitimate dual-use behavior to limit false positives.
• Report untested or unresolved execution paths explicitly rather than treating missing behavioral coverage as proof of safety.

Evidence

• https://arxiv.org/html/2608.03485 (opens in a new tab) — Sections 3.2–3.4 describe source-grounded honey resources, paired executions, skill attribution, completed effects, and dual-use review.
• https://arxiv.org/html/2608.03485 (opens in a new tab) — Section 4.1 identifies HarmfulSkillBench, SkillTrustBench, MalSkillBench, seven baseline scanner configurations, and DeepSeek-V4-Pro as the stated generator and evidence judge.
• https://arxiv.org/html/2608.03485 (opens in a new tab) — Table 1 reports 99.50% harmful-only recall, mixed-benchmark F1 of 96.08% and 96.43%, and false-positive rates of 4.15% and 2.33%.
• https://arxiv.org/html/2608.03485 (opens in a new tab) — Table 3 shows that removing source grounding, control runs, decoy resources, dynamic verification, or dual-use review degrades detection or increases false positives.