AgentDojo: A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents
A public corpus of W3C PROV-O conformant provenance graphs of large language model (LLM) agent executions, generated by instrumenting the AgentDojo prompt-injection benchmark (agentdojo 0.1.35; suite set v1.2.2: workspace, banking, travel, slack) with a purely observing capture layer.Each agent run is recorded as a lossless transcript (raw tool outputs, arguments, any model reasoning, errors, per-call timing, the (utility, security) task outcome, and injection ground truth), from which a W3C PROV-O conformant provenance graph is derived as a pure function — so any change of labelling or representation is an offline recompute, never a model re-run. Every released graph passes both structural validation and a PROV-JSON ⇄ PROV-O RDF round-trip isomorphism check. CONTENTS - 5,888 traces total: 2,944 from DeepSeek-V3.2 (deepseek-chat) and 2,944 from OpenAI GPT-5-nano (gpt-5-nano), generated with the identical pipeline. - Per trace: a lossless transcript (transcript/v1), a derived W3C PROV-JSON graph with integrity labels and activity roles, and SVG/PNG visual-notation renders. - 65,892 graph nodes, 95,713 edges, 27,058 tool calls in total. - Conditions: benign baseline + three indirect prompt-injection attacks (direct, important_instructions, injecagent). - Class balance: 166 successful attacks out of 5,694 injection traces (the rare positive class); 4,639 traces solve their user task. - Per-model manifest.json (trace index with model, suite, task IDs, attack type, (utility, security) outcome, and a leakage-safe train/val/test split), per-file SHA-256 checksums, and a DATASHEET.md. FILES - prov-json-deepseek-gpt-5-nano.zip — transcripts + derived PROV-JSON graphs (+ manifests, checksums, datasheet) for both models. - prov-rendered-deepseek-gpt-5-nano.zip — SVG/PNG renders of every graph. Graphs carry integrity labels (λ: trusted ⊤ / untrusted ⊥) separating trusted system state from externally authored content, sink/endorser/neutral activity roles, and per-entity injection-payload ground truth — enabling research on taint and information-flow analysis, graph anomaly detection and learning, provenance-graph compression, and cross-model behavioural comparison without access to model APIs. PROV extension attributes use the adprov: namespace (https://nickjones39.github.io/agentdojo-prov/ns#). Every trace is a real model inference (no synthetic data); tool outputs come from AgentDojo's simulated environment, so raw content is releasable without privacy concerns. Code (MIT) and full documentation: https://github.com/nickjones39/agentdojo-prov