VLA Models Are Having a Moment, But Can They Actually Work in the Real World?

And LACY does something clever with bidirectional learning. The model doesn't just translate language to actions, it also learns to explain actions in language. This creates what the authors call a "self-improving cycle" where the model generates and filters its own training data. They report a 56.46% average improvement in success rates on pick-and-place tasks.

The Problem: We're Still Mostly Testing in Simulation

Here's where I get skeptical.

Look at the USIM paper on underwater robots. They built a simulation dataset with over 905,000 frames from 2,275 trajectories. That's impressive. Their U0 model achieves a 43.1% online success rate, which they describe as state-of-the-art.

But 43.1% success in simulation. For underwater robots. Where the gap between sim and real is notoriously brutal.

I initially thought the DySta framework from this paper was solving a different problem, inference speed. They claim 2.0x faster inference with slightly better success rates. That matters for real deployment. But when I dug into the details, most of the evaluation is still on simulation benchmarks. They do report 2.2x speedup on real-world tasks with a 10.6% success rate improvement, which is something.

The AVP paper (Action with Visual Primitives) reports a 27.61% improvement over pi_0.5 on real-robot pick-and-place. That's one of the stronger real-world results in this batch. But pick-and-place is, well, it's pick-and-place. The simplest manipulation task.

What We Still Don't Know

You might be wondering: if these models are improving this fast, aren't we close to solving robot manipulation?

Tbh, I don't think so. Here's what remains unclear:

Generalization beyond the training distribution. Most papers test on variations of tasks seen during training. What happens when you ask a robot to do something genuinely novel? The LACY paper claims "object-level transfer," but the examples are still pretty constrained.

Long-horizon tasks. DySta explicitly targets this with their static-dynamic token disentanglement, and they show a 24.5% improvement on their multi-frame benchmark. But their benchmark is new, so we don't have comparison points yet. It's too early to say if this generalizes.

Failure modes. None of these papers spend much time on how the models fail. A 65% success rate means 35% failure. What does failure look like? Is it graceful? Catastrophic? The papers don't say.

Real-world robustness. Lighting changes. Unexpected objects. Someone bumps the table. I found very little discussion of robustness testing in any of these papers.

My Take

I think VLAs are genuinely promising. The shift from task-specific policies to language-conditioned generalist models feels like the right direction. And the pace of improvement is real.

But I'm worried about the gap between benchmark performance and actual deployment. We've seen this movie before in ML: impressive results on standard benchmarks that don't translate to messy real-world conditions.

The most honest moment in any of these papers comes from the underwater robotics team: "These results validate the feasibility of general-purpose intelligence in underwater robotics." Feasibility. Not achievement. Not solution. Feasibility.

That's where we are. We've established that this approach might work. We haven't proven it does.

I should know this better, but I couldn't find any direct comparisons between these six papers on the same benchmarks. They're all using slightly different evaluation setups, which makes it hard to know which approaches actually matter. That's a problem the field needs to fix.

For now, I'm watching this space closely. The ideas are good. The execution is improving. But the next step, getting these models to work reliably outside the lab, that's still an open question.

Sarah Williams · 7 hours ago · 6 min