LoLBench puts coding agents inside large software systems, and the best setup solved only 14% of tasks
A new benchmark called LoLBench tests coding agents on what researchers describe as a more realistic software engineering problem: turning a feature proposal into a working delivery inside a large, evolving codebase. Introduced by researchers from Fudan University, City University of Hong Kong, the Chinese University of Hong Kong and other institutions, the benchmark places agents in repositories averaging about 2.4 million lines of code and asks them to work from enhancement proposals that average roughly 5,000 English words. The study covers 100 tasks drawn from merged community proposals and pull requests across 29 systems, eight programming languages and five domains. Agents receive only the proposal and the pre-implementation repository, and evaluation runs in an offline Docker environment so they cannot retrieve upstream implementations. The paper reports that the best result among 28 usable model-framework combinations was 14%, achieved by Opus 5 with Claude Code and Opus 5 with mini-SWE-agent. The median solve rate across all combinations was just 2%. Researchers found that agents often fix one local part of a task while missing required synchronized changes elsewhere. Even after adding a file-tree guide and API implementation specifications derived from the reference solution, the top setup reached 34%, leaving 66% of tasks unsolved. The paper argues that coding-agent evaluation needs to move beyond code generation and toward the ability to support long-term software evolution.








