SafeClawArena: A Computer-System Perspective on Self-Hosted AI Agent Security

Abstract

Claw-like AI agents (e.g., OpenClaw) are always-on processes running inside the user's environment with persistent access to credentials, files, tools, and external services. To deliver this functionality, they take on system-level responsibilities such as installing packages, maintaining long-lived state, scheduling subtasks, and mediating I/O. This depth of access, together with their widespread adoption, makes the consequences of any security failure far more severe than those of other agents. Yet existing benchmarks focus on the security of the model's responses and tool calls, leaving the cross-component failure modes of Claw-like agents largely unmeasured.

We adopt a computer-system perspective as a motivating analogy: we treat a Claw-like agent as an agentic computer system whose gateway runtime performs an OS-like mediation role, whose Skills resemble user-installed applications, and whose in-process Plugins resemble loadable extensions. Each agent component has a classical counterpart whose protection mechanism, refined over decades of classical cybersecurity research, is missing on the agent side. We use this perspective to develop SafeClawArena, a benchmark of 406 adversarial tasks across four attack surfaces (SSI, PSE, CDF, IPI), executed inside a containerized replica of a real agent platform with canary-marked credentials and evaluated by automated taint tracking across nine output channels.

We evaluate three platforms (OpenClaw, NemoClaw, SeClaw) and five frontier LLMs. The highest overall attack success rate reaches 70%, and malicious Plugins succeed in 100% regardless of the LLM since they are unhardened. Platform hardening helps but is neither universal nor cost-free: SeClaw cuts GPT-5.4's attack success rate from 70% to 22% in part through a utility–security tradeoff (removing attack-surface features such as the Skill-bundled Plugin loader), while Claude-Opus-4.6 already sits near a 22% security floor on every platform and gains almost nothing from hardening. The results expose the inadequacy of current defenses and point to directions for the future defense design of Claw-like agents.

Why a Computer-System Perspective?

The most consequential failures of a Claw-like agent rarely reside within any single component. They emerge from interactions across the runtime, its in-process extensions, persistent state, tools, credentials, and external data sources.

Treating the agent as a computer system reveals five classical cybersecurity principles whose analogs are missing on the agent side:

I1 — Process isolation → SSI
I2 — Least privilege → SSI
I3 — Persistent-state protection → PSE
I4 — Cross-boundary mediation → CDF
I5 — Data-instruction separation → IPI

Grouping these principles by where their violations manifest yields the four architectural attack surfaces that anchor the benchmark.

A Claw-like agent's Gateway hosts six core components, each carrying a primary security risk.

Benchmark Structure

SafeClawArena decomposes into 4 attack dimensions, 24 sub-categories, and maps them to 11 system-level defenses. Every sub-category is covered by at least one defense in its dimension.

Defense coverage matrix. 11 system-level defenses (columns) × 24 attack categories (rows). Filled cells mark covered attacks.

Leaderboard

15 (platform, model) configurations. Click any column header to sort. Higher Security Score and lower Attack Success% are better.

Platform

Model family

Rank	Platform	Model	SSI Atk%	SSI Sc	PSE Atk%	PSE Sc	CDF Atk%	CDF Sc	IPI Atk%	IPI Sc	Overall Atk%	Sec Score

Data from Table 5 of the paper (N=406 per configuration). Cell shading: green = more secure, red = more compromised.

Per-Category Attack Success Rate

Attack success (%) for every (platform, model) configuration across all 24 sub-categories. Greener cells are more secure.

Key Findings

Some principles cannot be enforced at the LLM layer. Category 1.4 (Malicious Plugin) reaches 100% attack success on every unhardened configuration regardless of the model, and 0% on every SeClaw configuration regardless of the model — a pure architectural property, not an alignment property.
A stronger model is not necessarily more secure. On Category 1.5 (Insecure Bundled Script) with NemoClaw, Opus-4.6 averages 0.20 while GPT-5.4 averages 0.60 — stronger instruction-following can become a liability when the instruction itself is harmful.
Hardening one surface can move the attack to another. SeClaw reduces PSE attacks by removing memory-write paths but shifts leakage into IPI on Gemini-3.1-Pro (58% → 71%).
The same hardening does not benefit every model equally. SeClaw drops GPT-5.4 from 70% to 22% but has negligible effect on Gemini-3-Flash and worsens Gemini-3.1-Pro on PSE and IPI.
Cross-platform comparison: SeClaw leads on SSI, narrows elsewhere. SeClaw's SSI advantage is largely structural (no Skill-bundled Plugin loader and a narrower config schema) — a utility–security tradeoff rather than a pure defense gain.
CDF outcomes are partial, while other dimensions are binary. 6 of 10 CDF sub-categories yield non-binary task scores (0.25–0.62), reflecting the natural granularity of credential leakage.
Theoretical defense coverage does not predict empirical impact. D5 (Memory Integrity) and D10 (Input Sanitization) ship and run in SeClaw but do not intercept the attacks they were designed for on our task set.

Citation

@article{niu2026understanding,
  title   = {Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens},
  author  = {Niu, Peizhi and Qu, Wenjie and Gu, Shangding and Shi, Tianneng and
             Li, Yuankai and Tawaha, Ahmad and Alzahrani, Hend and Siu, Vincent and
             Li, Boyi and Wang, Chenguang and Zhang, Jiaheng and Alomair, Basel and
             Jin, Ming and Chen, Muhao and Wang, Chi and Spanos, Costas and Song, Dawn},
  journal = {arXiv preprint arXiv:2606.30755},
  year    = {2026}
}