The Real Problem With Robot Vision Isn't the Eyes, It's the Brain

The SceneDiver approach is what they call "coarse-to-fine focus plan generation," and honestly the idea is pretty sensible. Instead of trying to process everything at once, the system builds a scene graph first (basically a map of what's where), then progressively narrows down to the relevant bits. It's how humans actually look at cluttered environments, you don't process every object simultaneously, you scan, identify what matters, then focus.

The paper claims this "substantially reduces visual hallucinations" while keeping things fast enough for reactive control. I'd want to see more real-world testing before I got too excited, but the architecture makes sense.

Speed versus accuracy, the eternal tradeoff

The third paper, T-FunS3D, tackles a related but distinct problem: current methods that try to segment functional parts of a 3D scene are either slow and accurate or fast and useless. Their words, basically, are that existing approaches "attempt to segment the entire scene exhaustively, making them highly resource-intensive and time consuming."

Their solution is to be lazy in a smart way. Instead of segmenting everything, the system takes a task description, figures out which objects in the scene are actually relevant, and only then does the detailed work of identifying functional components. It's task-driven rather than scene-driven.

The results on the SceneFun3D benchmark are, they claim, comparable to state-of-the-art while being faster and using less memory. That's the kind of tradeoff that actually matters for deployment. A robot that takes 30 seconds to figure out where to grab something isn't useful in most applications.

So what

Here's my read on all this, and call me old-fashioned, but I think we're still in the "defining the problem correctly" phase rather than the "solving it" phase. All three papers are essentially building better benchmarks and evaluation frameworks as much as they're building better algorithms. A2A-Bench, SceneFun3D, the various metrics they're proposing, this is infrastructure work.

That's not nothing! One of the persistent issues in robotics research is that everyone tests on their own setups with their own metrics, which makes it basically impossible to compare approaches meaningfully. If these benchmarks get adopted, that's genuine progress even if the algorithms themselves get superseded in six months.

The deeper question, the one none of these papers really answers, is whether the current paradigm of vision-language models plus clever prompting and post-processing is going to get us where we need to go. The SceneDiver paper in particular notes that VLMs are "better at long-term planning" while VLAs (vision-language-action models) are "better at reactive control," which suggests we might need fundamentally different architectures for different aspects of the problem.

I don't know if that's right. Nobody does yet. But at least people are being honest about the limitations, which is more than I can say for some of the hype I see coming out of well-funded startups.

What happens next

All three teams say they're releasing code and datasets publicly, which is good. The A2A team specifically mentions an "agent-assisted annotation pipeline" that combines language model filtering with human verification, which could be useful for other researchers trying to build similar benchmarks without spending years on manual labeling.

The practical upshot for anyone actually trying to deploy manipulation systems is probably this: don't trust your VLM to figure out task-relevant regions on its own, at least not yet. The gap between "can identify objects" and "can identify what to do with objects given a task" is wider than the marketing materials suggest.

Will these specific approaches matter in two years? Hard to say. The field moves fast, and a lot of what seems promising in papers doesn't survive contact with real-world deployment. But the problems they're addressing, affordance grounding, visual hallucination, computational efficiency, those aren't going away. Someone's going to have to solve them eventually.

If you want to argue about any of this, my email's on the about page. I still check it, unlike certain messaging platforms the kids seem to prefer.

James Chen · 4 hours ago · 4 min