VLA Models Can't Actually Understand What You're Asking Them to Do

Look, I've seen enough spec sheets to know that demo success rates don't translate to real capability. But this is worse than I expected. These models have billion-parameter language backbones. They should, in theory, know that 7 × 8 = 56. The problem is that knowledge doesn't survive the translation to action.

Why does semantic grounding fail?

The paper points to a fundamental tension in how VLAs are trained. Robot fine-tuning optimizes for imitation over task-specific action distributions. In plain terms: the model learns to copy what worked in the training data, not to reason about what should work given the instruction.

This creates shortcuts. If most training examples show the robot grabbing the object closest to the gripper, the model learns "grab nearby things" rather than "grab the thing matching the instruction." The language instruction becomes noise that the model learns to ignore.

The researchers call these "visual or instruction-action shortcuts." I'd call them the predictable result of optimizing for the wrong objective. Imitation learning rewards matching demonstrated actions. It doesn't reward understanding why those actions were demonstrated.

This explains why VLA models can look impressive on standard benchmarks while failing spectacularly on RoboSemanticBench. Standard benchmarks often have implicit structure that shortcuts can exploit. A benchmark designed specifically to break shortcuts reveals the underlying gap.

Are researchers addressing this?

Several recent papers suggest the field is at least aware of the problem, even if solutions remain unclear.

A paper on Continuous Reasoning for VLA argues that natural language is "mismatched to the granularity of continuous control." Their proposed solution: replace discrete language tokens with continuous reasoning latents that can be shared across model instances and verified through downstream action improvement. On LIBERO-PRO, they report a 40.4% improvement in mean subtask success over π0.5 on one robot platform and 26.3% on another.

That's an ambitious number, and I'd want to see independent replication before getting too excited. But the framing is interesting. The authors argue that "reasoning in VLA is less about extra tokens than about a shareable, verifiable internal language for action." In other words, the problem isn't that models lack reasoning capability. It's that the reasoning doesn't connect to the actions.

Another paper, TIC-VLA, tackles a related problem: the temporal mismatch between slow semantic reasoning and fast reactive control. Their solution explicitly models reasoning latency during training, so the policy learns to compensate for the delay between understanding an instruction and acting on it. They report robust performance under "multi-second reasoning latency," which sounds impressive until you realize that multi-second delays are the baseline for current VLAs, not an extreme stress test.

What about zero-shot capability?

Most VLA evaluations report results after task-specific fine-tuning. This makes it hard to know whether the pretrained model learned anything useful or just provided a better starting point for task-specific memorization.

Wall-OSS-0.5 tries to answer this directly. It's a 4B parameter open-source VLA trained on over one million robot trajectories across 20+ embodiments. The key claim: the pretrained checkpoint achieves "non-trivial zero-shot real-robot behavior" before any task-specific fine-tuning, including on a held-out deformable manipulation task.

After fine-tuning, they report 60.5% average task progress on 15 real-robot tasks, outperforming π0.5 by 17.5%. The paper also claims that action training doesn't erode vision-language competence, which would address one common concern about VLA architectures.

I'm cautiously optimistic here. The open-source release means others can verify these claims, and the explicit focus on measuring pretrained capability (rather than post-fine-tuning performance) is methodologically sound. But 60.5% task progress still means nearly 40% failure, and we don't know how much of that success relies on the same shortcuts that RoboSemanticBench exposes.

Do world models help?

A separate line of research asks whether predicting the future (world modeling) leads to better robot behavior than directly predicting actions (VLAs).

A diagnostic study comparing World-Action Models (WAMs) and VLAs found that "success alone hides key differences." WAMs often improve object-level behavior and target selectivity, but gains depend heavily on architecture and come with higher inference costs.

The most interesting finding: sequential WAMs show the clearest predictive structure in their internal representations, while auxiliary and joint WAMs either compress or entangle future information in ways that may not help downstream control. This suggests that how you integrate world modeling matters as much as whether you include it.

The paper introduces a diagnostic framework that characterizes internal representations as "memorized, reactive, or predictive." Memorized features recall training data. Reactive features respond to current observations. Predictive features encode future-oriented structure. Only the last category should actually help with generalizable control.

Can you add explicit condition monitoring?

One pragmatic approach sidesteps the semantic grounding problem entirely: use a separate model to monitor task conditions and feed discrete directives to the VLA.

CLAW demonstrates this for weight-aware grasping. A fine-tuned CLIP model monitors a scale's digital readout and produces directives based on weight thresholds. These prompts feed into π0, which handles the actual visuomotor control. The decoupling lets CLAW satisfy precise numeric constraints that end-to-end VLAs struggle with.

This is basically an admission that current VLAs can't reliably ground numeric constraints into actions. The solution: don't ask them to. Use a vision-language model for the understanding part and a VLA for the action part.

It works, but it's architecturally inelegant. You're running two models, adding latency, and creating potential failure points at the interface. The real question is whether this is a temporary workaround or a permanent architectural pattern.

What does this mean for deployment?

The honest answer: it's too early to say. The gap between semantic competence and action grounding is real and well-documented now. Whether it's a fundamental limitation or a training problem remains unclear.

Some observations:

None of these are production-ready. The field is still figuring out what the problem actually is, let alone how to solve it.

From my time in hardware, I know that capability gaps like this tend to persist longer than researchers expect. The demo-to-deployment gap in industrial robotics took decades to close, and that was for systems with far simpler perception requirements. VLAs are attempting something much harder: grounding open-vocabulary language into continuous control.

The optimistic read: these benchmarks and diagnostic tools will accelerate progress by forcing the field to confront the gap rather than paper over it with cherry-picked demos.

The pessimistic read: semantic grounding might require architectural innovations we haven't discovered yet, and current VLA approaches might be fundamentally limited by their imitation learning foundations.

I'd bet on something in between. The next generation of VLAs will probably close some of the gap through better training objectives and reasoning architectures. But the idea that you can throw a billion-parameter language model at robot control and get human-level instruction following? That's still vaporware.

Mark Kowalski · 9 hours ago · 6 min