Real-SWE tests coding agents on private enterprise code, with the top pass rate reaching only 38.8%

Specific Labs, an AI data company backed by Y Combinator, has launched Real-SWE, a programming benchmark designed to test coding agents against real private codebases from operating companies. The tasks come directly from production environments and include tax calculation, billing migration, API billing, and customer data migration. Because neither the code nor the answers are publicly available online, agents have to work out each company’s business rules and code structure on their own. In the benchmark results shared in the report, Fable 5.1 ranked first with a 38.8% pass rate. GPT-6 Astra followed at 33.8%, while Gemini 3.8 Flash posted 31.2%. GLM 5.3 came in fourth at 28.8%, outperforming Grok 4.6, Kimi K3, and GPT-5.6 Sol. Among the 10 tasks currently disclosed, six had overall pass rates below 15%, and one task saw every model fail. According to Xiaopu Peng, a researcher at Zhipu, the team has been training long-horizon tasks since GLM-5.1. He said Real-SWE is closer than public SWE leaderboards to measuring that type of capability, which may help explain GLM 5.3’s stronger showing in this benchmark.