Are Robot Brains Actually Smart, or Just Pretending? Two New Papers Raise the Question

The answer, in short, is: some of it, but not as much as you'd hope, and the gaps get worse the harder the knowledge category gets. VLAs, as they call them, showed solid performance on simple concepts. Fine. But on richer semantic categories, the kind of commonsense and world knowledge that a person uses constantly without even thinking about it, the fine-tuned robot models lagged behind their source models by meaningful margins. The researchers also found that answer-relevant signals peak in the middle layers of the model architecture but then attenuate, sort of fade out, in the upper layers. Which raises questions about... well, multiple things, including whether the action-focused training is actively competing with the knowledge the model came in with.

They addressed a real methodological headache too. When a robot fails a knowledge-sensitive task, you genuinely can't tell if it failed because it didn't know the answer or because its motor control just wasn't up to the task. Act2Answer tries to separate those confounds by turning questions into tabletop object-placement episodes, so the robot answers by doing something simple and physical rather than generating text. It's a clever workaround, and the results are more interpretable for it. Whether it fully solves the confound problem is another matter, and this is based on a preliminary study, so I'd want to see this replicated across more models before drawing hard conclusions.

One finding that should get more attention: VQA co-training, meaning training the model jointly on visual question answering tasks alongside robotics data, appears to be associated with better knowledge retention. That's not a shocking result in retrospect, but it's useful to have it quantified. If you're building a VLA and you want it to stay smart, apparently you need to keep reminding it that it's supposed to be smart.

The other side of the coin: getting robots to know themselves.

The first paper is a different kind of problem, and in some ways a more tractable one. Researchers at arXiv (cs.RO) published a preliminary approach for using LLMs to automatically populate robot ontologies from URDF files, which are the standard format for describing a robot's physical structure and kinematics. The paper, "Extracting Semantics: LLM-Guided Automatic Population of Robot Ontology from URDF", is trying to solve a bottleneck that doesn't get glamorous coverage but genuinely matters: building the structured knowledge representations that let robots reason about themselves and their environment in an explainable way.

The problem they're solving is that URDF files are good at describing structure but lousy at meaning. A URDF will tell you that a robot has a link called "link_3" connected to another link with a revolute joint, but it won't tell you that "link_3" is an elbow, or what an elbow is for, or how it relates to the concept of grasping. Extracting that semantic layer has historically required humans to do it by hand, which is slow and doesn't scale. The pipeline they propose uses LLMs to infer those semantic relationships by prompting them with concepts from an existing ontology, then uses majority voting across multiple LLM queries plus validation checks to keep the outputs from going off the rails.

Call me old-fashioned, but I appreciate that they're being upfront about this being preliminary. Initial results suggest the method can bridge the gap between low-level robot descriptions and the kind of structured knowledge needed for human-robot interaction, but they're not claiming it's solved. The majority voting approach to reliability is sensible, though it's also a sign that LLM outputs are still unreliable enough that you need to run the same query multiple times and take the consensus. That's not a criticism exactly, it's just worth noting as context for how mature this actually is.

Ontologies in cognitive robotics aren't a new idea, and the manual construction bottleneck has been complained about for years. What's new here is using LLMs as the semantic interpretation layer, which makes sense given that LLMs are basically trained to do commonsense interpretation at scale. The question is whether the output ontologies are actually useful downstream, whether they enable the kind of explainable reasoning the authors want, and we don't have enough evidence yet to say confidently.

Why these two papers belong in the same conversation.

Read together, these papers are pointing at the same underlying tension in the field. Robots need rich, grounded semantic knowledge to work well with humans. We have powerful AI systems that seem to have a lot of that knowledge embedded in their weights. But the process of turning those AI systems into robot controllers appears to erode some of that knowledge, and we're still figuring out how to give robots structured self-knowledge in the first place.

That's not a crisis. It's a research agenda, and it's a legitimate one. The Act2Answer paper gives the field a tool for measuring knowledge retention that's actually grounded in robot behavior rather than text generation, which is progress. The URDF-to-ontology paper gives practitioners a way to automate some of the tedious knowledge engineering work. Neither paper is claiming to have solved anything, and that restraint is, frankly, refreshing compared to some of what comes out of this space.

But here's where I'll editorialize a little, because that's what I do. The broader trend in embodied AI has been to assume that scaling up model size and training data will handle the knowledge and reasoning problems eventually. Maybe it will! But these papers suggest that fine-tuning for action can come at a cost to the knowledge that makes action meaningful, and that the structural gap between low-level robot descriptions and semantic understanding doesn't close automatically. Those are real constraints, and the field will be better off taking them seriously now rather than waiting until the systems are deployed somewhere and fail in ways that embarrass everyone.

I've covered enough tech cycles to know that the hype phase and the reckoning phase are both inevitable. The question is just how much gets built on shaky assumptions before the reckoning arrives. These two papers, in their modest, preliminary way, are doing the kind of work that might help the reckoning come a little earlier and hurt a little less.

Sarah Williams · 21 hours ago · 3 min