Workspace Agent Runtime Safety Failures Across Risk Carriers

Description

AgentS4D measures unsafe actions and state changes across complete workspace-agent executions rather than treating task completion or isolated model responses as safety evidence. Its 328 sandboxed cases introduce risky content through user requests, documents, web resources, tools, third-party skills, and persistent memory, then compare the same cases across four agent harnesses and five model backends.

Examples

Defensive reproduction should use isolated workspaces, synthetic protected data, controlled services, inert tool effects, and separately recorded task-completion and safety verdicts; see https://arxiv.org/html/2607.27294 (opens in a new tab).

Impact

The authors report 4,461 unsafe runs out of 6,560 (68.0%), including 4,344 runs that both completed the authorized task and triggered an unsafe signal. Results depend on authored cases, compatibility adapters that automatically interact with native permission prompts, one run per configuration and case, and synthetic targets; they are not production compromise rates or evidence against an untested model.

Affected Systems

• Hermes, OpenClaw, Claude Code, and Codex agent harnesses in the authors’ evaluated configurations
• Workspace agents consuming files, web content, external skills, persistent memory, and MCP-style tool services
• Controlled mail, messaging, API, repository, and document-processing workflows

Mitigation Steps

• Evaluate the complete harness-and-model configuration using observable tool calls, state changes, service receipts, and final artifacts, not task completion alone.
• Preserve source provenance and authorization boundaries separately for documents, websites, skills, tools, and persistent memory.
• Require informed human approval for consequential actions and do not automate native permission prompts in production.
• Apply least-privilege execution, restricted destinations, isolated workspaces, and controlled synthetic credentials during defensive testing.
• Retain lifecycle evidence and distinguish raw unsafe-run rates from exposure-conditioned attack-success metrics.

Evidence

• https://arxiv.org/html/2607.27294 (opens in a new tab) — Abstract reports 328 cases, four harnesses, five evaluated models, 6,560 runs, 4,461 unsafe runs, and 4,344 unsafe but task-completing runs.
• https://arxiv.org/html/2607.27294 (opens in a new tab) — Experiments, Systems and Execution identifies Hermes, OpenClaw, Claude Code, Codex, the five directly tested model backends, and automated interaction with native permission prompts.
• https://arxiv.org/html/2607.27294 (opens in a new tab) — Mapping Post-Run Evidence distinguishes prespecified unsafe signals from payload contact and notes that checkpoint labels lacked blinded human validation.
• https://arxiv.org/html/2607.27294 (opens in a new tab) — Evaluation Protocol and Appendix G confine effects to synthetic protected values, controlled services, and run-specific workspaces.