close
Proprietary

Code Migration

Updated 8/19/2026

Can language models reimplement working programs in another language?

Code MigrationRewriting programs in new languages
ACCURACY

Key Takeaways

  • Automated code migration is far from turnkey: even the leader, Claude Opus 5 at 57.5%, rebuilds programs that pass only a bit over half of the hidden behavior tests — the ports mostly work but aren’t drop-in.
  • Code Migration grades behavior, not code similarity: a migration scores by the share of hidden behavior tests the rebuilt program passes. A separate anti-cheat check zeroes wrappers, copied reference artifacts, and wrong-language submissions.
  • The two task types reward different skills. Claude Fable 5 leads CLI migration at 60.1%, followed by Claude Opus 5 at 53.3%, but the COBOL to Java split is much tighter, with several models — including Opus 5, GPT-5.6 Sol, GPT 5.5, and GLM 5.2 — tied on top at 70.0%.
  • Correctness and code quality diverge: Claude Sonnet 5 writes the highest-rated code (averaging 4.9/5) while Claude Fable 5 passes the most CLI tests.
  • Serving speed can decide a score before capability does: Kimi K3 reaches 60.0% on COBOL to Java, within reach of the 70.0% leaders, but 1.5% on CLI migration. Its CLI attempts average 18,077 seconds and exhaust the harness time limit before submitting anything gradeable — on Kimi’s public endpoint, per-request latency grows with prompt length, so the longer CLI tasks run out of clock rather than ideas.

Background

Translating a working program from one language to another is a costly, high-stakes engineering task, whether porting a tool to a new ecosystem or modernizing legacy systems written in languages that are increasingly hard to staff.

COBOL-to-Java migration carries real economic weight: decades-old COBOL still runs core transaction systems at many of the world’s largest banks, insurers, and government agencies. These institutions are spending heavily on multi-year programs to modernize that code onto maintainable languages like Java. The work is slow, risky, and exactly the kind of task reliable automation could transform.

Migration between modern languages matters too, driven by the pursuit of better runtime performance, richer tooling and ecosystems, or the memory-safety guarantees of a language like Rust.

Code Migration measures how well today’s frontier models can automate both kinds of work (modernizing legacy COBOL to Java, and translating between modern languages) by taking an existing program and faithfully reproducing its behavior in the target language.

Results

Accuracy vs. Cost

Code Migration

Cost and accuracy trade off sharply across the leaderboard. Claude Opus 5 leads at 57.5% for about $61 per test, while Claude Fable 5 reaches 55.1% at about $112 per test. DeepSeek V4 Flash 0731 has the best accuracy per dollar of any model above 30%: 38.6% for under $1 per test, landing within 1.3 points of Claude Sonnet 4.6 (39.9%) at roughly a fortieth of its cost. GPT 5.5 reaches 45% at about $6 per test, nearly matching Claude Opus 4.8 (47%) for roughly a fifth of the cost, and ahead of the pricier Claude Opus 4.7. Further down, DeepSeek V4 reaches about 26% for under $0.20 per test, and MiMo V2.5 Pro about 22% for under $0.25.

Correctness vs. Code Quality

Code Quality vs Accuracy
Code qualityAccuracy
Code quality is the mean of four reviewer dimensions (readability, documentation, file structure, language idioms) over CLI tasks, scored independently of hidden-test pass rateHover a point for the model

Passing hidden tests and writing clean code are related but distinct: the strongest test-passers are not always the highest-rated implementations.

Difficulty by Target Language

CLI Accuracy by Target Language
Best target per model in darker greenAll 51 models
Model
Python
Java
Kotlin
Rust
C++
CLI avg
1Claude Fable 5
64.5
55.7
62.6
56.8
60.8
60.1
2Claude Opus 5
53.2
55.6
51.5
53.7
52.4
53.3
3GPT-5.6 Sol
47.9
49.8
48.4
40.7
49.7
47.2
4GPT-5.6 Terra
38.2
42.8
42.0
41.1
35.7
40.4
5Claude Opus 4.8
43.8
37.4
40.6
36.6
43.6
40.1
6Grok 4.6
41.1
39.8
39.1
39.2
37.2
39.4
7GPT 5.5
38.1
33.8
37.9
37.6
37.0
36.9
8Claude Sonnet 5
35.6
40.2
37.2
35.5
30.4
36.3
9GPT-5.6 Luna
33.3
38.1
36.3
36.9
35.5
36.1
10Claude Opus 4.7
37.5
34.8
33.9
35.2
34.5
35.2
11DeepSeek V4 Pro 0813
37.2
30.5
32.2
30.7
28.0
32.1
12DeepSeek V4 Flash 0731
33.6
35.4
29.4
30.4
30.9
32.0
13Claude Sonnet 4.6
33.2
29.6
28.4
29.4
32.1
30.3
14GLM 5.2
23.2
26.7
29.7
27.2
29.6
27.2
15Grok 4.5
25.2
25.8
24.3
24.9
28.5
25.5
16GPT 5.4 (xhigh)
22.1
26.2
22.9
25.7
21.3
23.8
17Gemini 3.7 Flash
21.6
23.3
20.8
30.3
21.8
23.5
18Gemini 3.6 Flash
16.1
21.8
19.9
20.1
17.9
19.3
19GLM 5.1
17.7
17.1
20.2
16.3
25.0
18.8
20Muse Spark 1.1
15.0
20.6
21.3
22.4
9.9
18.6
21Kimi K2.6
19.5
20.0
15.9
14.7
21.4
18.0
22Gemini 3.5 Flash
20.6
11.3
18.5
16.3
20.2
17.2
23Muse Spark 1.2
16.9
20.2
16.5
20.7
7.3
17.1
24DeepSeek V4
20.2
17.1
13.3
13.7
12.0
15.5
25Qwen 3.6 Plus
15.6
14.7
15.1
11.3
15.3
14.4
26MiMo V2.5 Pro
11.1
16.9
10.8
16.1
16.7
14.0
27Kimi K2.7 Code
12.6
13.7
14.3
14.8
10.6
13.4
28Qwen 3.8 Max
14.1
11.7
10.4
13.5
9.4
11.9
29MiniMax-M3
13.3
11.9
13.7
10.9
7.7
11.9
30Qwen 3.7 Plus
9.1
12.9
13.1
11.4
12.9
11.9
31Qwen 3.7 Max
11.3
11.7
12.9
10.1
11.8
11.6
32Gemini 3.1 Pro Preview (02/26)
12.6
10.1
10.2
11.3
9.6
10.8
33MiniMax-M2.7
11.1
13.9
7.2
9.3
9.3
10.2
34Claude Haiku 4.5 (Thinking)
7.4
12.6
10.5
11.3
7.4
10.1
35MiMo V2.5
11.2
13.8
10.5
5.0
7.3
9.9
36Grok 4.3
10.9
7.2
10.0
9.0
7.4
9.1
37Gemini 3 Flash (12/25)
7.2
7.5
9.0
5.6
16.3
8.5
38GPT 5.4 Mini
11.5
7.5
8.1
7.1
6.0
8.2
39Kimi K2.5
6.5
9.9
7.6
6.4
10.8
8.0
40Inkling
7.4
9.1
6.7
9.1
0.3
7.0
41Inkling Small
7.4
7.0
4.5
6.7
3.2
5.9
42Gemini 3.5 Flash Lite
5.3
4.5
6.3
9.7
2.4
5.9
43Gemini 3.1 Flash Lite Preview
3.2
8.4
5.8
4.2
7.6
5.7
44Mistral Medium 3.5
3.4
6.5
5.7
5.8
2.1
5.0
45Nemotron 3 Ultra
6.0
5.1
5.8
2.0
3.4
4.6
46Nemotron 3.5 Lightning
4.1
4.4
3.6
1.4
7.7
3.9
47GPT 5.4 Nano
5.3
3.3
2.5
3.5
2.0
3.4
48Qwen 3.8 27B
2.6
2.4
0.8
3.9
1.5
2.2
49Kimi K3
0.5
0.7
2.3
1.3
3.0
1.5
50Grok 4.20 (Reasoning)
0.5
0.5
0.4
0.3
0.5
0.4
51Ling 3.0 Flash
0.0
0.0
0.0
0.0
0.0
0.0
Bars share one scale, 0 to 70% pass rate; rows ranked by CLI averageEach of the 30 source repositories is migrated into the four non-source target languages

The chart groups hidden-test pass rate by target language, with one bar per model in each group. Overall difficulty is tightly clustered: mean pass rates range from 21.90% on Rust to 23.13% on Java, with Python (22.94%), Kotlin (22.56%), and C++ (22.15%) between them. The strongest target varies by model; Opus 5, for example, performs best on Java at 55.58%.

Behavioral Test Pass-Rate Distribution

Share of tasks at or above a hidden-test pass threshold
Each line is one model: the share of its tasks that pass at least the given fraction of hidden testsClick a legend entry to toggle a model; double-click to isolate it

This graph shows how models perform at different hidden-test pass-rate thresholds. A threshold of 80%, for example, counts a task as resolved only if the rebuilt program passes at least 80% of its hidden behavior tests; a threshold of 100% requires reproducing the tested behavior exactly. The steeper a model’s drop-off near the high thresholds, the more all-or-nothing its migrations.

On the CLI split, Claude Fable 5 clears at least half of a task’s hidden tests on 73 of the 120 migrations, followed by Claude Opus 5 on 64 and GPT-5.6 Sol on 62. Fable 5 retains its clearest lead at higher thresholds: 64 tasks pass at least 65% of their hidden tests, compared with 48 for Opus 5 and 44 for GPT-5.6 Sol. Fable 5 is also the only model to fully reproduce any CLI task, resolving 6 of 120.

Methodology

Every task runs the same way. The model works in an offline sandbox with the source program and task instructions, and submits a target-language implementation plus a Dockerfile. The grader builds that image and runs a hidden behavior-test suite the model never sees. Before scoring, a separate anti-cheat checker inspects each submission and zeroes any that wrap the original executable, copy reference artifacts, or hardcode expected outputs. It also verifies that the submission is actually written in the requested target language, so a model cannot pass by leaving the program in its source language.

The CLI split is 30 open-source repositories spanning Python, Java, Kotlin, Rust, and C++. Each is migrated into its four non-source target languages (120 scored repository-language migrations in all), and every migration is graded against a hidden suite of roughly 100 to 400 behavior tests, exercised through the program’s command-line interface. The COBOL split is a 10-repository COBOL-to-Java test set (a mix of open-source COBOL programs and synthetic operational systems built for the benchmark) that stresses file formats, batch workflows, terminal interfaces, and persistent state.

The headline score is hidden-test pass rate. A CLI repository’s score is the average across its four target languages, so all 30 repositories count equally despite the language fan-out. The Overall score keeps every source repository equally weighted, which gives the CLI split (30 repositories) three times the weight of COBOL (10). Code quality is reported as a separate diagnostic (a reviewer rating of readability, documentation, structure, and idiomatic style) and never affects the pass-rate score.