LoLBench puts coding agents inside large software systems, and the best setup solved only 14% of tasks

LoLBench puts coding agents inside large software systems, and the best setup solved only 14% of tasks

N
News Editor
2026-11-04 12:20:44
A new benchmark called LoLBench tests coding agents on what researchers describe as a more realistic software engineering problem: turning a feature proposal into a working delivery inside a large, evolving codebase. Introduced by researchers from Fudan University, City University of Hong Kong, the Chinese University of Hong Kong and other institutions, the benchmark places agents in repositories averaging about 2.4 million lines of code and asks them to work from enhancement proposals that average roughly 5,000 English words. The study covers 100 tasks drawn from merged community proposals and pull requests across 29 systems, eight programming languages and five domains. Agents receive only the proposal and the pre-implementation repository, and evaluation runs in an offline Docker environment so they cannot retrieve upstream implementations. The paper reports that the best result among 28 usable model-framework combinations was 14%, achieved by Opus 5 with Claude Code and Opus 5 with mini-SWE-agent. The median solve rate across all combinations was just 2%. Researchers found that agents often fix one local part of a task while missing required synchronized changes elsewhere. Even after adding a file-tree guide and API implementation specifications derived from the reference solution, the top setup reached 34%, leaving 66% of tasks unsolved. The paper argues that coding-agent evaluation needs to move beyond code generation and toward the ability to support long-term software evolution.

Coding agents can already call tools, edit files and run tests. What they still struggle to do is take a new requirement and turn it into a complete feature inside a large, evolving software system. That is the problem targeted by LoLBench, a benchmark introduced by researchers from Fudan University, City University of Hong Kong, the Chinese University of Hong Kong and other institutions.

LoLBench puts coding agents inside large software systems, and the best setup solved only 14% of tasks 2

The benchmark places agents in software systems averaging about 2.4 million lines of source code and evaluates the full path from a feature proposal to module delivery. According to the paper, the main failure point is not that agents cannot change code at all, but that they may update one local area correctly while missing other modules that also need coordinated changes. Even with added implementation guidance, most tasks remained unsolved.

The paper is available at https://arxiv.org/abs/2609.37143, and the dataset is hosted at https://huggingface.co/datasets/lolbench26/LoLBench.

How LoLBench defines long-horizon coding work

The researchers argue that long-horizon coding should not be defined only by elapsed time, interaction rounds or token usage. An agent that prints one number per minute from 1 to 1000 may take a long time, but the task is still just a simple loop. In the same way, local bug-fixing benchmarks such as SWE-bench do not capture the full challenge of moving from requirement analysis to module delivery.

LoLBench puts coding agents inside large software systems, and the best setup solved only 14% of tasks 3

LoLBench focuses on module development in complex systems. After receiving a new requirement, a developer usually has to work through several questions: what behavior must be implemented, whether the current architecture can support it, which dependencies will be affected, and how the result should be verified. The benchmark brings that hidden reasoning process into evaluation and asks the agent to infer an implementation plan from user intent on its own.

The paper splits the challenge into two parts. The first is perceptual complexity: understanding the proposal, identifying implicit constraints and cross-module dependencies, and inferring the modification scope and implementation specification. The second is implementation complexity: turning that specification into correct cross-file edits, then testing and repairing the system without breaking existing behavior.

100 tasks across 29 systems, with proposals averaging 5,000 words

LoLBench selects 100 tasks from community enhancement proposals and their corresponding merged pull requests. Those tasks span 29 systems, eight programming languages and five domains. During evaluation, the agent receives only the proposal and the repository state before implementation. The proposals average about 5,000 English words. Testing runs in an offline Docker environment to prevent the agent from retrieving upstream implementations.

One example in the paper is PEP 617, which proposed replacing CPython's LL(1) parser with PEG. Implementing that change requires work across grammar definitions, parser generation, the abstract syntax tree and error reporting. The agent has to identify that full implementation chain.

LoLBench puts coding agents inside large software systems, and the best setup solved only 14% of tasks 4

On perceptual complexity, the average mention rate of target class names from the reference implementation inside the proposal text is only 7.3%. That is less than half of SWE-bench Verified at 15.9%, and less than one-fifth of FeatureBench at 38.7%. In other words, the proposal often does not point directly to the code that needs to change.

Average implementation touches 5,499 lines of code

On implementation complexity, repositories in LoLBench average about 2.4 million lines of source code, roughly 9.4 times the size of SWE-bench Verified and 4.9 times the size of FeatureBench. The average implementation changes 5,499 lines, compared with 38 lines and 2,050 lines in those two benchmarks.

Each task changes an average of 64.9 files, 40.1 classes and 195.9 functions. The file count is 25 times that of SWE-bench Verified and 4.2 times that of FeatureBench. The paper says this makes cross-module coordination and verification much harder.

LoLBench uses two kinds of tests to check whether delivery is correct. Feature-to-pass, or F2P, requires tests to fail before implementation and pass after it. F2P checks behavior through public interfaces, which allows different internal implementations. Pass-to-pass, or P2P, requires tests to pass both before and after the change.

LoLBench puts coding agents inside large software systems, and the best setup solved only 14% of tasks 5

On top of each project's original tests, the researchers built mutant implementations that omit functionality or tamper with requirements. They then added test cases to catch errors that would otherwise slip through and to raise coverage of the reference patch. After that enhancement, the detection rate across 1,688 mutants rose from about 61% to 96%, while average line coverage for F2P by task increased from about 60% to 79%.

Best solve rate among 28 agent combinations was 14%

The team evaluated 28 usable combinations built from six frontier models and five scaffolds, with a six-hour limit per task. The top result was a tie: Opus 5 with Claude Code and Opus 5 with mini-SWE-agent both reached a 14% solve rate. The median across all combinations was only 2%.

Using the same model, Opus 5, mini-SWE-agent matched Claude Code on solve rate, but took 3.2 times longer and generated five times as many tokens. The extra cost did not produce a better outcome.

LoLBench puts coding agents inside large software systems, and the best setup solved only 14% of tasks 6

Token usage also shows where the effort goes. Averaged across agents, the localization stage accounted for 52.4% of non-cached input tokens and 45.4% of generated tokens. By interaction rounds, localization also consumed more turns than other stages.

Compared with the files and functions modified in the reference implementation, the 28 agent setups posted file-edit precision ranging from 38.0% to 80.4% and recall from 30.3% to 65.0%. Function-edit precision ranged from 54.2% to 65.6%, while recall ranged from 20.8% to 53.4%. The paper's conclusion is that code localization remains inaccurate and incomplete.

Extra guidance helped, but most tasks still failed

The researchers then kept the original feature proposal and added two more pieces of information derived from the reference implementation: a modified file tree and API implementation specifications. The goal was to help the agent identify the scope of changes. Under that setup, Opus 5 with Claude Code improved from 14% to 34%, GPT-5.6 Sol with Codex rose from 3% to 25%, and DeepSeek V4 Flash with OpenCode increased from 1% to 17%.

Even then, the best combination still left 66% of tasks unsolved. The paper says the added guidance reduces perceptual burden, but agents still have to coordinate modules, implement interfaces and pass verification. That points to implementation complexity in large systems as a separate obstacle, not just a search problem.

LoLBench puts coding agents inside large software systems, and the best setup solved only 14% of tasks 7

From code generation to software evolution

The results suggest several concrete directions for agent development: whether repository retrieval can cover key dependencies, whether implementation specifications are complete before coding starts, and whether verification can catch cross-module omissions early enough.

The researchers write that the value of software engineering lies in whether user requirements can be delivered reliably. Agents need to understand why a change is happening, how it affects the system and how to implement it completely. In that framing, LoLBench is meant to push evaluation beyond code generation and toward support for continuous software evolution in large systems.

The paper also cites PEP 617 at https://peps.python.org/pep-0617/. The article source says the original Chinese write-up came from the WeChat public account "Xinzhiyuan," authored by Xinzhiyuan and edited by LRST.

This article was originally published by Bit.Fan. For more cryptocurrency news and market insights, visit www.bit.fan.
8700

Disclaimer:

The market information, project data, and third-party content displayed on this platform are for industry information sharing only and do not constitute any form of investment advice or return commitment.

Cryptocurrency trading carries high risks. Users should fully assess their risk tolerance and make independent decisions. All profits, losses, and legal responsibilities are borne by the users themselves.

LoLBench puts coding agents inside large software systems, and the best setup solved only 14% of tasks | Bit.Fan