OpenAI reports 87% coding benchmark pass rate across 20 programming languages

New evaluations show gains in AI-generated software and COBOL migration, including languages still widely used in banking, scientific computing and essential services

OpenAI has published new coding evaluations measuring AI model performance across 20 programming languages and the migration of existing COBOL software

OpenAI has reported substantial improvements in its AI models' ability to write and modernize software in less commonly used programming languages, with GPT-6 Astra achieving an 87.4% average test pass rate across 20 languages, compared with 26.8% for GPT-5.6 Sol.

The results come from MultiLangBench, a new evaluation developed by OpenAI to test whether its models can build working software across different programming languages. A separate assessment examined their ability to migrate existing COBOL programs into newer languages without changing how those programs behave.

The work focuses partly on languages such as COBOL, Fortran and Visual Basic, which have less publicly available code and documentation for AI models to learn from than widely used alternatives such as Python and JavaScript.

Despite their age, these languages remain embedded in essential infrastructure. According to figures cited by OpenAI, COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions and 95% of ATM transactions. Fortran continues to be used in applications including weather forecasting and nuclear reactor safety analysis.

Modernizing these systems presents particular challenges when documentation is incomplete, original developers are no longer available and business rules exist only within the code itself.

GPT-6 Astra records gains across all 20 languages tested

Rather than asking AI models to complete isolated coding exercises, MultiLangBench requires them to build a SQLite-compatible database engine from scratch in a specified programming language.

The models must organize code, manage information, handle errors and introduce functionality while preserving what already works. OpenAI then compares the resulting database files against its reference SQLite implementation.

At maximum reasoning effort, GPT-6 Astra achieved an average test pass rate of 87.4%, with improvements recorded in every language tested.

Some of the largest gains came in languages where the previous model had struggled:

• COBOL improved from 10.9% to 72.1%.

• Fortran increased from 21.4% to 83.5%.

• Visual Basic rose from 27.5% to 89.4%.

• OCaml improved from 25.6% to 85.8%.

Astra exceeded 83% in 19 of the 20 languages, with COBOL remaining the most difficult.

OpenAI also tested GPT-6 Luna as a lower-cost alternative. It achieved an average pass rate of 40.3%, compared with 26.8% for GPT-5.6 Sol, at an estimated average API cost of $0.77 per sample rather than $3.48.

That represents an estimated 78% reduction in cost for the lower-cost model comparison, although Luna's overall test performance remains considerably below Astra's.

COBOL migration accuracy reaches 97.9% in separate evaluation

OpenAI's second assessment examined a different problem: whether AI can translate existing COBOL programs into C++, Java, Kotlin, Python or Rust without altering their underlying behavior.

This involves more than producing code that runs successfully. In financial systems, for example, differences in rounding rules or how missing records are handled can change the results of a transaction even when the rewritten software runs without errors.

To test this, OpenAI used hidden assessments requiring the migrated programs to produce byte-for-byte identical observable outputs. A migration counted as successful only when every test passed.

GPT-6.1 Sol achieved 86.6% accuracy at low reasoning effort, compared with 5.4% for GPT-5.6 Sol at the same setting, while reducing estimated API costs by 39%.

At higher reasoning effort, GPT-6.1 Sol reached 97.9% accuracy.

The figures represent performance on OpenAI's own evaluations, rather than demonstrated success across production banking or other critical systems. The company acknowledges that deploying migrated software still requires expert review and testing against existing business rules and dependencies.

OpenAI is releasing the MultiLangBench evaluation setup, test suite and language-level results on GitHub, allowing developers to inspect the tests and examine performance in languages relevant to their work.

Its latest models are available through the API for coding tasks, with the company continuing to seek developer and enterprise feedback on languages and workflows requiring further improvement.

ETIH Innovation Awards 2027 open for entries

The ETIH Innovation Awards 2027 are now open for entries, with 26 categories recognising companies, institutions, projects and individuals making a measurable impact across education technology. The global awards span AI in education, digital learning, higher education, skills and workforce, inclusion, education operations, evidence and impact, with entries managed through ETIH’s new online awards platform.

Entries are open worldwide, with the early-bird deadline on Friday 18 December 2026 and the final entry deadline on Friday 26 February 2027 at 23:59 UK time. 

Explore the 2027 categories and judging criteria, read the How to Enter guide or start an entry through the ETIH Innovation Awards 2027 platform.