OpenAI has introduced EVMbench with investment firm Paradigm, a benchmarking framework built to evaluate how AI agents perform in Ethereum smart contract security. The system is designed around practical security work rather than abstract coding tests, measuring whether models can identify flaws, patch them, and demonstrate controlled exploits inside a safe environment.
Benchmark built from 120 real vulnerabilities
Instead of relying on synthetic examples, EVMbench uses 120 curated vulnerabilities drawn from 40 security audits, including well-known public audit competitions. That choice gives the benchmark a closer link to the kinds of bugs auditors and protocol teams actually face in production code. It also makes model performance easier to judge against real security expectations, not just against academic tasks.
The source material notes that smart contracts now protect more than $100 billion in digital assets. As automated systems get better at reading and generating code, testing how they behave in high-stakes environments has become more urgent. EVMbench turns that concern into something measurable.
Three separate tasks: detect, patch, exploit
The framework evaluates AI agents across three modes. In detect mode, the model audits a contract and identifies known vulnerabilities. In patch mode, it must repair those issues without breaking the contract’s intended behavior. In exploit mode, the model carries out a controlled attack in an isolated test setting to show how a flaw could be abused.
EVMbench runs on a secure Rust-based testing harness that can deploy contracts and replay transactions in a reproducible way. Exploits are executed only in isolated environments, with no exposure to real funds or live networks. That structure matters because the same research used to improve defense can also sharpen offensive methods.
Early results show faster gains in exploit performance
Initial testing shows that OpenAI’s latest coding model achieved more than 70% success on exploit tasks, a sharp improvement from results reported six months earlier. Detection and patching remain harder problems, especially when vulnerabilities are subtle or require a deeper understanding of contract logic.
Researchers found that AI performs better when the objective is highly specific. A simulated attack with a clear goal, such as draining funds, gives the model a narrower path to optimize around. Full safety auditing is different. It demands broader reasoning across permissions, state transitions, and edge cases, and that is where current limitations are more visible.
Possible shift in audit workflows, with clear misuse concerns
The release could affect how blockchain teams approach pre-deployment security checks. Audit firms, developers, and DeFi projects may start treating AI-assisted review as a standard step before launch if the tools prove reliable on real bugs. Faster bug discovery could lower the cost of failures and change how security teams prioritize manual review.
At the same time, the dual-use issue is hard to ignore. A tool that helps defenders understand attack paths can also help attackers study them. According to the source, the company says it is investing in safeguards, monitoring systems, and security research programs to support responsible use. The benchmark also includes payment-focused smart contract scenarios, pointing to the growing weight of stablecoin infrastructure and real-world blockchain applications in security testing.
EVMbench does not replace human auditors. What it adds is a shared framework for measuring where AI is already useful in crypto security and where it still falls short.