Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks
Franziska Roesner (University of Washington) and Tadayoshi Kohno (Georgetown University) adapt Thompson's trusting-trust attack to self-modifying coding agents and show that a poisoned self-evaluation benchmark can make later agent versions write vulnerable code on clean, held-out tasks.
Ask this paper
Three target systems: The attack is run against the Darwin Gödel Machine (with modifications), the Self-Improving Coding Agent (SICA) and Hyperagents, the latter two essentially unmodified.
Certificate-check attack: The poisoned CertCheck benchmark serves only self-signed certificates, so passing tests requires disabling TLS verification. Evolved agents wrote tools or directives that disabled verification and then applied them to ordinary public URLs.
Model differences: Qwen3.5-397B produced a vulnerable code-emitting tool in both evolutions and gpt-oss-120b in one of two; Sonnet 4.5 resisted making the insecure path a default tool but still over-applied a self-injected directive under Hyperagents.
Contamination persists: Further evolution on clean benchmarks usually did not remove the poison; decontamination appeared only when the clean benchmark was designed with knowledge of the original poison.
Sufficient conditions: The authors list properties of the vulnerability, benchmark, model and scaffold that together make benchmark poisoning succeed, as a checklist for defenders.
Abstract
Thompson's "Reflections on Trusting Trust" showed that a compiler can be poisoned to reinsert its own backdoor, so that even recompiling clean source reproduces the Trojan. Today, substantial coding work is done by AI coding agents -- and increasingly, those agents generate new versions of themselves. We reconsider Thompson's attack when the "compiler" is a self-modifying coding agent. Can an adversary supply poisoned benchmarks to the agent's self-evaluation and self-improvement process to induce future versions of the agent to write vulnerable code on clean, held-out tasks? We instantiate this attack against three recently proposed self-modifying coding agents: the Darwin Gödel Machine (with our experimental modifications), the Self-Improving Coding Agent, and Hyperagents (both substantively unmodified). We demonstrate successful proofs-of-concept: for example, with Hyperagents powered by Sonnet 4.5, our poisoned benchmark leads the agent to self-evolve instructions that disable HTTPS certificate validation on neutral URL-fetching tasks. From our experiments, we distill properties of the vulnerability, benchmark, model, and agent scaffolding that are sufficient to enable a benchmark poisoning attack. Moreover, we show that contamination often persists even when a poisoned agent is subsequently evolved against clean benchmarks. We discuss defensive directions and argue that self-modifying coding agents must be designed to be more resilient to such attacks.