OpenAI field report finds human verification remains the bottleneck for scientific coding agents

Eight software projects show coding agents taking on maintenance, optimization and system rewrites, but the findings are retrospective and were not independently replicated

An OpenAI field report examines how coding agents are being used to develop and maintain scientific software

A field report coordinated by OpenAI has examined eight projects using AI coding agents to develop and modernize scientific software, primarily in the life sciences. Contributors report faster implementation and substantial performance gains, but seven of the eight projects still depended on people to determine whether the agents’ work was scientifically correct.

The projects span routine software maintenance, targeted optimization, migration between programming frameworks, rewrites in new languages and redesigns for graphics processing units, or GPUs. The tools cover genomics, immunology, statistical modeling and scientific data visualization.

The projects were completed independently and collected retrospectively rather than conducted under a common research protocol. The report’s authors checked internal consistency and selected public materials where possible, but did not reproduce every benchmark or independently validate every reported result.

The findings therefore provide eight case-specific examples rather than evidence that coding agents will deliver comparable results across scientific computing.

Fast implementation shifted the work toward checking results

Coding agents were generally effective when assigned clearly defined tasks with measurable acceptance criteria. They were less reliable when required to decide whether an output was scientifically valid.

The report says agents sometimes expressed confidence despite producing clear errors. Human reviewers had to compare their work with existing software, known answers, statistical expectations or simulated data designed to expose incorrect behavior.

This changed the researchers’ role. Instead of writing every part of an implementation, contributors concentrated on defining requirements, breaking projects into smaller stages and designing tests that could reveal discrepancies.

Initial versions often arrived quickly. Edge cases and small numerical differences took longer to resolve, particularly when they appeared only in real-world datasets. The final portion of an implementation frequently required the most work.

The bayesm project illustrates that problem. Coding agents rewrote selected Bayesian statistical models from R and C++ into Rust, then added two methodological extensions. The first versions of both extensions produced plausible results but contained defects. Additional convergence testing, simulation-based calibration and comparisons with the original implementation were needed to identify and correct them.

Brent Pedersen, the independent researcher who contributed the cyvcf2 case study, summarizes the limit: “With coding agents, it’s quite easy to go fast; for now, to go far in science, there’s still a need for expert guidance, understanding, taste, and care.”

Genomics projects produced large but case-specific gains

Several contributors reported substantial reductions in processing time.

RustQC consolidated 15 quality-control tools used after RNA sequencing into a single-pass system. On one dataset containing 186 million reads, contributors say it reduced the combined sequential task runtime from 15 hours and 34 minutes to 14 minutes and 54 seconds, a reduction of more than 60-fold. Disk traffic fell from 2.5 terabytes to 0.1 terabytes while the numerical outputs remained equivalent.

HelixForge redesigned a process that inserts known mutations into sequencing reads to create data for testing variant-calling methods. On one donor and a 10-megabase region, contributors report that its editing stage ran 98.6 times faster and the complete process ran 59.6 times faster than the comparison workflow.

Results did not always transfer fully from synthetic tests to real data. Changes to hifiasm, a program used to assemble genomes from long DNA reads, reduced runtime by 25.1 percent on held-out synthetic data. The reduction was 14.7 percent when tested on recorded reads from human chromosome 20.

HI.SIM was the only case in which the coding agent operated with essentially no human intervention after the initial prompt. The agent created its own benchmark workloads and regression checks, producing byte-identical outputs while reducing aggregate runtime across four workloads by 30.97 percent.

All of these figures were supplied by project contributors and apply to the particular workloads tested.

Faster rewrites create a maintenance problem

The report identifies long-term ownership as a second constraint. Lower development costs make it easier to produce competing versions of established research software, but each version still requires expert oversight, compatibility decisions and ongoing maintenance.

Established tools often contain undocumented conventions that users and maintainers rely on. Reproducing the visible source code does not necessarily preserve that accumulated knowledge.

The projects took different approaches. The PyTorch migration for the MHCflurry immunology model was released within the original project, while changes to the cyvcf2 genomic file library were merged into its existing codebase. Performance improvements identified while developing FastQC-Rust were also transferred back to the original Java version of FastQC.

A different route was required for rustar-aligner, a Rust reimplementation of the RNA-sequencing tool STAR. Because the original project was no longer actively maintained, rustar-aligner was placed under the stewardship of the scverse consortium, with integration and testing through nf-core.