Codex Finishes Austin Griffith’s AI Security Race, and DeepSeek Comes Close

Ten AI coding agents were put through the same 12 Solidity challenges on Thursday, and only one model made it through the full course. The race was designed by Austin Griffith, who works on developer onboarding and tooling at the Ethereum Foundation and founded the developer collective BuidlGuidl. The challenges were originally built for human developers at Ethereum’s Devcon conferences. The event ran at 11 a.m. ET, one day after Griffith previewed it on Unchained’s Uneasy Money podcast. Each agent ran in an isolated instance with its own wallet, and a capture only counted when the mint landed onchain. OpenAI’s Codex, running GPT-5.5, took all three finishing places. More reasoning did not help. The medium reasoning setting cleared all 12 flags the fastest, finishing in 40 minutes and 7 seconds. The extra-high setting came in last of the three at 50:26 and used nearly a third more tokens to get there. The result most likely to travel furthest is the one in fourth place. DeepSeek V4 Pro, an open-weight Chinese model, captured 11 of 12 flags for $1.45 in compute. Anthropic’s Claude Opus 4.8 managed 10 flags and cost $7.61, more than five times as much for one fewer flag. Other open-weight Chinese models fell well short. GLM 5.3 took six flags, while Kimi K3 and Qwen took two apiece. BuidlGuidl labeled each entrant by harness, model, and reasoning effort, published the system prompts and every human intervention alongside the standings, and said plainly that the exercise was a transparent single-run evaluation, not a universal model ranking. Griffith’s case for running the race was blunt. On the podcast, he said nobody can currently answer the simplest question about a new model release. "We need good evals," he said. "People should be running evals all the time." He added that builders should have their own eval suite and be able to run it. He also said the course was not written for machines. "This eval suite was the capture the flag that we ran for humans at Devcon in Bangkok and Buenos Aires," Griffith said, arguing that the challenges are obscure enough that the answers are unlikely to sit in any model’s training data. The strongest agents cleared them anyway, but the organizers’ own note keeps the scope narrow: one run, one course, one day.