OpenAI's wet lab claims deserve scrutiny: what the GPT-5 biology papers actually show

The second paper is more of a methodological proposal. It argues that benchmarking AI on biology requires moving beyond static datasets to actual wet lab experiments. The case study involves molecular cloning, where GPT-5 was used to optimize a protocol. The paper acknowledges both "promise and risks" of this approach, though it's light on specifics about what those risks entailed in practice.

What's genuinely new here

To be fair to OpenAI, there is something novel in this work. Most AI-for-science benchmarks rely on retrospective evaluation: you train a model, test it on held-out data, and report metrics. The problem is that held-out data from past experiments may not reflect the actual distribution of problems a working scientist faces. It's worth noting that this critique isn't new (several groups have made similar arguments), but OpenAI is among the first major labs to publish results from prospective, closed-loop experiments with a frontier model.

The Ginkgo collaboration is also genuinely interesting as an engineering achievement. Connecting a language model to real lab automation, handling the inevitable failures and edge cases, and running the system long enough to get meaningful results is harder than it sounds. Anyone who's worked with automated biology platforms knows they break constantly. The fact that this worked at all suggests serious integration effort.

What's incremental (or unclear)

Here's where I'm going to be picky, and I know I'm being picky here, but the details matter.

The 40% cost reduction figure is presented without adequate context. Cost reduction compared to what baseline? The papers describe the starting protocol as "standard," but standard protocols vary enormously across labs. A 40% improvement over a poorly optimized starting point is very different from 40% over a state-of-the-art protocol developed by experienced researchers.

More importantly, we don't know how much of the improvement came from GPT-5's reasoning versus simple search. If you give any optimizer enough experimental budget to try variations, it will find improvements. The question is whether the model is doing something smarter than random or grid search. The papers don't include ablations comparing GPT-5 to simpler baselines (Bayesian optimization, for instance, which has been used for protocol optimization for years).

The "autonomous" framing also deserves skepticism. Actually, the research shows the system required significant human scaffolding: defining the search space, setting up the automation, interpreting failures, and presumably intervening when things went wrong. This is closer to "AI-assisted optimization" than "autonomous research," and the distinction matters for understanding what the technology can actually do.

The evaluation framework problem

The second paper's proposal for wet lab evaluation is conceptually reasonable but methodologically thin. The core idea, that we should test AI on real experiments rather than just benchmarks, is correct. But the paper doesn't grapple seriously with the challenges this creates.

Wet lab experiments are expensive, slow, and noisy. Running enough experiments to get statistically meaningful comparisons between AI systems would cost millions of dollars and take months. The paper doesn't propose solutions to this, it mostly just acknowledges the problem exists.

There's also a reproducibility concern. If the evaluation requires access to Ginkgo's specific automation platform, other researchers can't replicate or extend the work. This is a general problem with industry AI research, but it's especially acute for work that claims to be establishing evaluation standards.

I'd want to see, at minimum, detailed protocols that could be run on other automation platforms, or ideally on manual benchtop setups. Without that, this is less an "evaluation framework" and more a case study.

The biosecurity question

Both papers mention dual-use concerns in passing, but neither engages seriously with the implications. If GPT-5 can optimize protein synthesis protocols, it can presumably optimize other biological processes too. The papers don't describe what safeguards were in place, whether the model was restricted from certain types of optimization, or how OpenAI plans to handle these risks as capabilities improve.

This isn't hypothetical hand-wringing. Cell-free protein synthesis is a dual-use technology; it's used for legitimate research but could also be used to produce harmful proteins. The papers are silent on whether the optimization was constrained to avoid certain directions, or whether the resulting protocols were reviewed for safety before publication.

It remains unclear whether OpenAI has a systematic framework for evaluating these risks, or whether biosafety review happened on an ad-hoc basis for these specific papers.

What this means for biology research

Let me be clear about what I'm not saying. I'm not saying this work is unimportant or that AI won't transform biology research. It probably will. The question is whether these specific papers represent that transformation or are better understood as early-stage proof of concept.

My read: this is solid engineering work that demonstrates language models can be integrated into automated biology workflows. The optimization results are real but modest, and we don't have enough information to know whether GPT-5 is doing something qualitatively different from existing optimization methods.

The more interesting implication is what this suggests about where AI-for-science is heading. If frontier labs are investing in wet lab integration and prospective evaluation, we'll likely see more work in this direction. That's probably good for the field, even if individual papers oversell their results.

Open questions

Several things remain unclear from these papers:

1. How does GPT-5's optimization compare to established methods like Bayesian optimization or evolutionary strategies? Without these comparisons, we can't assess whether the model is contributing genuine insight or just serving as a fancy search algorithm.

2. What was the total experimental budget? The papers don't disclose how many experiments were run, making it impossible to evaluate efficiency.

3. How much human intervention was required? The "autonomous" framing obscures the actual human-AI collaboration structure.

4. What biosafety review process was applied? Given the dual-use nature of the technology, this seems like a significant omission.

5. Can these results be reproduced on other platforms? The reliance on Ginkgo's specific infrastructure limits external validation.

What I'd want to see next

If OpenAI (or others) want to make credible claims about AI accelerating biology research, here's what would be convincing:

First, rigorous comparisons to baseline optimization methods. Run the same experiments with Bayesian optimization, random search, and human experts. Show that GPT-5 does something these approaches can't.

Second, detailed methodology that enables replication. Publish the exact prompts, the automation scripts, the failure modes encountered. Make it possible for other labs to try this.

Third, honest accounting of costs and limitations. How much did these experiments actually cost? How many failed? What couldn't the system do?

Fourth, serious engagement with biosafety. If you're going to publish work on optimizing biological protocols, explain your risk assessment framework.

The current papers are interesting as early exploration, but they don't yet support the strong claims being made in the surrounding coverage. That's not a criticism of the research itself, which seems competently executed within its scope. It's a criticism of the gap between what the work shows and how it's being positioned.

Biology is hard. Automating biology is harder. And evaluating whether AI is genuinely helping requires the same rigor we'd apply to any other scientific claim. So far, the evidence is suggestive but incomplete.

Aisha Patel · 34 mins ago · 8 min