Modelcode Research / RepoMod-Bench
A benchmark for repository-scale code modernization.
RepoMod-Bench measures how well AI coding agents modernize entire repositories—not isolated files or functions. It uses the source repository as fixed ground truth and hidden behavioral tests to evaluate functional equivalence.
Performance collapses as repositories grow.
Across four evaluated agent configurations, the average percentage of hidden behavioral tests passed falls sharply with repository size.
| Codebase size | Average pass rate |
|---|---|
| Under 10,000 lines of code | 91.3% |
| Beyond 50,000 lines of code | 15.3% |
Fully autonomous code modernization at repository scale remains an open problem.
Real repositories. Hidden tests. Fixed ground truth.
Behavior is the contract.
RepoMod-Bench avoids judging whether an agent produced one preferred implementation. It asks whether the modernized repository still behaves like the source system.
- 01
Standardize the interface
Repositories expose consistent surfaces such as command-line or REST interfaces so source and target can be exercised the same way.
- 02
Hide the tests
Implementation-agnostic black-box suites remain unavailable to the agent, preventing test-driven shortcuts and overfitting.
- 03
Compare behavior
The original repository is fixed ground truth. The translated repository passes only when its observable behavior remains equivalent.
Repository scale changes the problem.
Many coding benchmarks test bounded tasks with visible tests. Modernization must preserve behavior across dependency graphs, build systems, interfaces, and thousands of interacting files.
RepoMod-Bench creates a shared measurement for that harder setting. Its results show that strong performance on small repositories does not establish reliable autonomy on large production systems.
RepoMod-Bench, briefly.
What is RepoMod-Bench?
A benchmark from Modelcode AI that measures how well AI coding agents modernize entire software repositories rather than isolated files or functions.
What are the headline results?
Average pass rates fall from 91.3% on repositories under 10,000 lines of code to 15.3% on repositories beyond 50,000 lines.
How is it different from other coding benchmarks?
It evaluates whole repositories with hidden, implementation-agnostic black-box tests. Agents are judged on functional behavior, not a particular implementation.
Who created RepoMod-Bench?
Xuefeng Li, Nir Ben-Israel, Yotam Raz, Belal Ahmed, Doron Serebro, and Antoine Raux at Modelcode AI.
Modernization needs more than agent autonomy.
Morph turns a large migration into an approved plan, reviewable milestones, and evidence your team can evaluate before merging.