SUSVIBES: Is Vibe Coding Safe?
Free while signed in. Answers cite the passages they came from.
Researchers introduce SUSVIBES, a benchmark of 200 real-world software engineering tasks to evaluate the security of code generated by LLM agents through "vibe coding" - the minimal-supervision programming paradigm. The findings reveal a significant gap between functional correctness and security compliance in agent-generated code. - **Benchmark design:** SUSVIBES contains 200 feature-request tasks from real-world open-source projects that, when given to human programmers, led to vulnerable implementations. This tests whether agents replicate common security mistakes. - **Alarming security gap:** While SWE-Agent with Claude 4 Sonnet achieved 61% functional correctness, only 10.5% of solutions met security standards. All evaluated coding agents performed poorly on security-sensitive tasks. - **Mitigation ineffective:** Preliminary mitigation strategies, such as providing vulnerability hints alongside feature requests, proved ineffective at improving security outcomes. - **Production risk:** The findings challenge optimism around LLM-assisted development, suggesting that widespread vibe coding adoption poses significant risks in security-critical applications.
Benchmark design: SUSVIBES contains 200 feature-request tasks from real-world open-source projects that, when given to human programmers, led to vulnerable implementations. This tests whether agents replicate common security mistakes.
Alarming security gap: While SWE-Agent with Claude 4 Sonnet achieved 61% functional correctness, only 10.5% of solutions met security standards. All evaluated coding agents performed poorly on security-sensitive tasks.
Mitigation ineffective: Preliminary mitigation strategies, such as providing vulnerability hints alongside feature requests, proved ineffective at improving security outcomes.
Production risk: The findings challenge optimism around LLM-assisted development, suggesting that widespread vibe coding adoption poses significant risks in security-critical applications.
Abstract
Vibe coding is a new programming paradigm in which human engineers instruct large language model (LLM) agents to complete complex coding tasks with little supervision. Although vibe coding is increasingly adopted, are its outputs really safe to deploy in production? To answer this question, we propose SU S VI B E S, a benchmark consisting of 200 feature-request software engineering tasks from real-world open-source projects, which, when given to human programmers, led to vulnerable implementations. We evaluate multiple widely used coding agents with frontier models on this benchmark. Disturbingly, all agents perform poorly in terms of software security. Although 61% of the solutions from SWE-Agent with Claude 4 Sonnet are functionally correct, only 10.5% are secure. Further experiments demonstrate that preliminary security strategies, such as augmenting the feature request with vulnerability hints, cannot mitigate these security issues. Our findings raise serious concerns about the widespread adoption of vibe-coding, particularly in security-sensitive applications.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack