The Factuality Problem Nobody Wants to Talk About

I initially thought this was a solvable problem. Train the models to be more uncertain, right? But after reading through both papers more carefully, I'm less optimistic. The issue seems baked into how these systems work. They're pattern matchers trained to sound confident because confident-sounding text is what they learned from.

Why are they releasing this now?

This is the part I keep coming back to. Both Google and OpenAI could have sat on these benchmarks. They make their flagship products look, well, unreliable. So why publish?

A cynical read: they're getting ahead of regulation. If you define the problem yourself, you get to control the narrative around solutions. A more charitable read: they genuinely want the research community to work on this.

I think it's probably both? Plus there's a competitive dynamic here. If your competitor releases a factuality benchmark and you don't, it looks like you're hiding something. So everyone releases at once.

What I find interesting is what's not in these benchmarks. Neither one really tests the kind of complex, multi-step factual reasoning that people actually use these models for. They're testing the simplest possible case, and the models are still struggling.

What this means for anyone building with LLMs

Okay, so what do you actually do with this information?

If you're building products on top of these models, the honest answer is: be very careful about use cases where factual accuracy matters. That sounds obvious, but I see startups every week pitching AI tools for legal research, medical information, financial analysis. Domains where being wrong 30-40% of the time isn't a quirky limitation, it's a liability.

The standard advice is retrieval augmented generation (RAG), where you ground the model's responses in verified documents. That helps. But it's not a complete solution. The model can still misinterpret the documents, or confidently extrapolate beyond what they say.

I should know this better, but I'm genuinely unclear on whether there's a technical path to solving this. The researchers I've talked to seem split. Some think it's a training data problem that more curation could fix. Others think it's fundamental to how transformer architectures work. We probably won't know for a few more years.

The uncomfortable implication

Here's what I keep thinking about. We're deploying these systems at massive scale, into search engines, into productivity tools, into educational software. Millions of people are getting factual information from models that, by their own creators' measurements, are wrong a third of the time.

And the failure mode isn't obvious. The model doesn't crash or throw an error. It just confidently tells you something false, and unless you already know the correct answer, you have no way to tell.

That's... a weird situation to be in. We've never had information sources that were simultaneously this fluent and this unreliable. Books can be wrong, but they're usually consistent. Wikipedia can be wrong, but it shows its sources. These models are neither consistent nor transparent about their uncertainty.

I don't have a clean conclusion here. The benchmarks are useful because they force us to confront the scope of the problem. But they don't tell us what to do about it. For now, I think the answer is just: verify everything important. Which sort of defeats the purpose of having an AI assistant, but here we are.

Maybe the next generation of models will be better calibrated. Maybe we'll figure out training techniques that reduce hallucination. Or maybe we'll just get better at building systems that work around the limitation. It's too early to say which path we're on.

What I do know is that 40% error rates on simple facts isn't good enough for most of the applications people are building. And I'm glad we finally have numbers to point to when making that argument.

Sarah Williams · 1 hour ago · 6 min