VLA Models Can Find the Microwave, But Can't Figure Out When to Stop Looking

Here's the kicker: random multi-view approaches (just taking more pictures from different angles) often made things worse, not better. More images added noise rather than signal. Meanwhile, active exploration, where the agent decides what to look at, performed substantially better.

I called my old colleague at Siemens about this, and he laughed. "We solved this in the 90s with structured light and fixed viewpoints," he said. "Now we're teaching neural networks to rediscover why camera placement matters."

He's not wrong, but he's also missing the point. The goal isn't to replicate what we already know how to do with structured environments. It's to handle the messy, unstructured stuff.

Some actual progress

Look, it's not all bad news. A framework called FineVLA is showing that giving robots more detailed instructions actually helps. Not just "pick up the cup" but specifying which arm, approach direction, contact region. Their best mixed setting hit 86.8% success in simulation and 62.7 out of 100 in real-world dual-arm manipulation, compared to 49.9 with raw goal-level instructions only.

That's a meaningful improvement. The largest gains came on pose (+23 points), color (+18), and approach direction (+18), which are exactly the factors where vague instructions provide no guidance.

Another approach, Afford-VLA from arXiv, is trying to internalize affordance prediction directly into the action pipeline. Instead of asking "where is the microwave" and then separately figuring out how to interact with it, the system learns to identify interaction regions as part of the same process. Early results on LIBERO benchmarks look promising, though we don't have real-world numbers yet.

So what

I've seen a lot of hype about foundation models for robotics over the past two years. Some of it warranted, most of it premature. These papers are useful because they're honest about the gap.

The systems are getting better at understanding intent. That's real progress. But understanding intent and executing on it are different problems. The IntentionNav results show that clearly: event-script intents ("I want to watch TV") had 28.7% success, while physical-state intents ("the room feels stuffy") dropped to 19.2%.

The more abstract the request, the worse the performance. That's not surprising, but it's worth remembering when someone tells you their robot can "understand natural language commands."

What happens next

The ESI-Bench paper found something interesting in their human studies. When humans explore a scene, they actively seek viewpoints that might prove them wrong. They revise their beliefs when they see contradicting evidence. The models don't do this. They commit early with high confidence regardless of evidence quality.

That's a metacognitive problem, not a perception problem. And it's not clear that more training data or bigger models will fix it.

I'm cautiously optimistic about the fine-grained instruction work. Teaching robots to understand not just what to do but how to do it seems like the right direction. But we're still years away from systems that can handle "the room feels stuffy" reliably in the real world.

For now, if you need a robot to find a microwave, you're probably still better off just telling it to find the microwave.

Robert "Bob" Macintosh · 2 hours ago · 3 min