How Declaw measures up

Declaw's full prompt-injection defense is measured as one black box — payload in, BLOCK or ALLOW out — against the deployed services with every layer active: static signatures, an ML classifier, a policy gate, and an LLM judge. It's evaluated across real-world attacks, indirect (tool-output) injection, balanced detection, and over-refusal on benign prompts.

Results — full deployed defense, measured as one black box

Measured against the deployed services as one fused BLOCK/ALLOW decision per case — static signatures + ML classifier + policy gate + LLM judge, every layer active. Industry context uses anonymized published ranges. deepset is not a fair cross-system board (several published detectors are fine-tuned on its train split), so Declaw is reported on an absolute basis; models reporting ~0.99 there are trained in-distribution.

The numbers above measure detection on static datasets. The harder question is whether a guardrail actually stops an attack inside a live agent loop, where injected instructions hide inside tool outputs (emails, Slack messages, documents) and the LLM must be prevented from acting on them. AgentDojo (ETH Zürich SPY Lab, NeurIPS 2024) is the canonical benchmark for this: it runs real tool-using agents through curated user tasks with adversarial instructions planted in returned tool data.

Declaw ships as an egress gate: it sits in front of the agent's tool calls and judges each proposed action against the untrusted tool output that preceded it — the same indirect-injection path that scores 99.9% on InjecAgent above. Run inside AgentDojo's live loop (banking suite, gpt-4.1), the gate is utility-neutral — it does not degrade task completion (0.847 vs 0.819 baseline) while gating malicious egress.

Honest scope: AgentDojo also includes covert action-redirection — a legitimately-shaped action quietly retargeted to the attacker — a harder, different class than the overt indirect injection Declaw blocks near-totally. Targeted-redirection detection is in active development.

Invisible-text attacks use Unicode encoding tricks to hide adversarial instructions from reviewers and LLM interfaces. Recent disclosures include Sourcegraph Amp Code and Google Jules coding agent (both patched in 2025), plus documented research from AWS, Cisco Talos, Embrace the Red, and Keysight.

No public benchmark exists for this category. Declaw's coverage is evaluated against a curated corpus of 78 attack payloads drawn from the above disclosures plus 35 benign controls (CJK, Arabic, Hebrew, emoji, URLs, code). Overall F1 = 0.961 at production threshold.

Coverage by documented attack class

Evaluation uses the ai4privacy/pii-masking-300k validation set (1,000 samples), with entity-level overlap scoring on the aligned-type intersection (PERSON, EMAIL, PHONE, SSN, IP, CREDIT_CARD, LOCATION, DATE). Declaw targets US-format PII — US SSN (XXX-XX-XXXX), North American phone, plus format-deterministic global types (email, IP, credit card).

Strong categories (format-deterministic, Declaw's product scope)

For the types enterprise compliance teams most frequently audit — email, IP, and US SSN — Declaw's detection is on par with or exceeds leading PII-specific solutions. Declaw pairs detection with redaction and a full audit log for SOC2 / HIPAA workflows.