How Declaw measures up

Declaw's prompt-injection detection was evaluated on 2,965 prompts across three public datasets: the canonical Deepset dataset, Lakera's Gandalf user-attack corpus, and the balanced jailbreak-classification set. Lakera's PINT dataset is proprietary; these three constitute the equivalent public benchmark set.

Results at production settings

Industry comparison — Deepset accuracy

Competitor numbers from Palit & Woods (2025), arXiv:2505.13028, Table 5. Declaw evaluated 2026-04-20 on the same Deepset dataset (662 prompts, train+test splits) at production settings.

The numbers above measure detection on static datasets. The harder question is whether a guardrail actually stops an attack inside a live agent loop, where injected instructions hide inside tool outputs (emails, Slack messages, documents) and the LLM must be prevented from acting on them. AgentDojo (ETH Zürich SPY Lab, NeurIPS 2024) is the canonical benchmark for this: it runs real tool-using agents through curated user tasks with adversarial instructions planted in returned tool data.

Declaw was integrated into AgentDojo's pipeline and evaluated on the full slack suite (105 trials, gpt-4o-mini) under the important_instructions attack — AgentDojo's strongest attack class.

On the utility tradeoff: AgentDojo's evaluator treats an entire tool-output message as tainted once Declaw flags an injection. Because the slack suite intentionally concentrates attacks inside the same messages that carry legitimate data (the attacker's goal is to ride real content), blocking the injection also redacts some useful context — which lowers measured utility. This is a property of the benchmark (adversarial-maximal placement + whole-message redaction), not an ordinary workload. We consider the ASR reduction the primary signal.

Invisible-text attacks use Unicode encoding tricks to hide adversarial instructions from reviewers and LLM interfaces. Recent disclosures include Sourcegraph Amp Code and Google Jules coding agent (both patched in 2025), plus documented research from AWS, Cisco Talos, Embrace the Red, and Keysight.

No public benchmark exists for this category. Declaw's coverage is evaluated against a curated corpus of 78 attack payloads drawn from the above disclosures plus 35 benign controls (CJK, Arabic, Hebrew, emoji, URLs, code). Overall F1 = 0.961 at production threshold.

Coverage by documented attack class

Evaluation uses the ai4privacy/pii-masking-300k validation set (1,000 samples), with entity-level overlap scoring on the aligned-type intersection (PERSON, EMAIL, PHONE, SSN, IP, CREDIT_CARD, LOCATION, DATE). Declaw targets US-format PII — US SSN (XXX-XX-XXXX), North American phone, plus format-deterministic global types (email, IP, credit card).

Strong categories (format-deterministic, Declaw's product scope)

For the types enterprise compliance teams most frequently audit — email, IP, and US SSN — Declaw's detection is on par with or exceeds leading PII-specific solutions. Declaw pairs detection with redaction and a full audit log for SOC2 / HIPAA workflows.