Self-Evolving Static Analysis via LLM Pseudo-Labeling
Rui Melo, Andre Duarte, Carolyn Rose and colleagues (Carnegie Mellon University, University of Porto) present SESAA, in which an LLM agent starts with no rules and iteratively writes joern queries for static analysis using only feedback from a second LLM critic.
Ask this paper
Method. The critic inspects flagged and unflagged code to identify likely false alarms and suspected misses without access to ground-truth labels, and the agent revises its queries from that feedback.
Benchmark result. On the OWASP Benchmark for Python, SESAA reaches a mean F1 of 0.742 over eight runs, above the default configurations of five widely used tools and comparable to the strongest of them after tuning with ground-truth labels.
Real CVEs. In a pilot on real Python projects, queries induced from code that predates a vulnerability detect 10 of 19 later CVEs.
Output. The detector is a set of readable queries that engineers can inspect, modify and run without the LLM.
Abstract
Security bugs often slip past static analysis not because they are impossible to detect, but because detection rules have not yet been written for the code in front of us. New frameworks, APIs, and coding idioms continually create such gaps, leaving developers to discover failures and manually encode new rules. We present SESAA, a framework in which an Large Language Model (LLM) agent starts from no rules and iteratively evolves a set of readable and auditable joern queries. The agent receives feedback only from a second LLM critic, which inspects flagged and unflagged code to identify likely false alarms and suspected misses, without access to ground-truth labels. The agent then uses this feedback to revise its queries. On the OWASP Benchmark for Python, SESAA reaches a mean F1 of 0.742 across 8 independent runs, well above the default configurations of five widely used tools and comparable to the strongest of them once it is tuned with access to ground-truth labels that SESAA never sees. In a pilot on real-world Python projects, queries induced from code available before a vulnerability was introduced detect 10 of 19 subsequent CVEs, including 9 detected by rules written for the same repository. The resulting detector is a set of executable, human-readable queries that engineers can inspect, modify, and run independently. These results suggest that LLM agents can turn static-analysis rule maintenance from a manually authored process into an iterative process of automated discovery and repair.