VLA Models Know More Than They're Taught, and Researchers Are Figuring Out How to Use It
A wave of new research suggests vision-language-action models encode information about success that was never part of their training objective. That's weird, and potentially very useful.
Crédit photo: Lottie animation by Centre Robotics (LottieFiles Free, used with credit). · source
Why do robot foundation models seem to know things nobody taught them?
I've been reading through a batch of recent papers on vision-language-action models, and honestly, I keep coming back to this one finding that I can't quite shake. Researchers at various institutions have been poking around inside frozen VLA representations, and they're discovering something strange: these models appear to encode information about whether they're succeeding at a task, even though their training loss never asked them to estimate success at all.
Let me back up. VLAs are trained through imitation. You show them demonstrations, they learn to copy the actions. That's it. The loss function cares about action prediction, not about whether the robot is making progress toward a goal or whether it's about to fail. And yet, when researchers from the arXiv study attached simple linear probes to frozen features from OpenVLA and Pi0.5, they could predict Monte-Carlo outcome targets with surprising accuracy. Pi0.5 probes hit roughly 92% pairwise ordering accuracy under same-task, same-timestep conditions. That's not nothing.
You might be wondering: is this just the model picking up on obvious cues? Like, maybe it's just learned that certain visual states correlate with success because they appear near the end of demonstrations? The researchers tried to rule that out by testing against baselines built on progress, time-to-go, and task identity. The success information was still there, and it was substantially more predictable than those alternatives. I initially thought this might be a clever artifact of the experimental setup, but after reading through their matched comparison methodology, I'm less skeptical.
À lire aussi
More in AI Models
The company just raised its outlook by a staggering amount, and honestly, I'm trying to figure out if this is real momentum or a peak we're about to fall off.
Sarah Williams · 2 hours ago · 5 min
A $65 billion raise that eclipses OpenAI. I've seen big valuations before, but this one's got me scratching my head.
Robert "Bob" Macintosh · 2 hours ago · 3 min
The private equity giants are seeking additional investors for what would be one of the largest AI infrastructure financing deals to date.
James Chen · 3 hours ago · 4 min
The company that once prided itself on vertical integration is outsourcing its AI brain to a competitor. That's not a pivot, it's a concession.
The practical payoff is interesting too. Using these probes as a test-time selector over sampled action prefixes, they pushed success rates on a push-plate task from 26.7% under greedy decoding to 44.3%. That's a meaningful jump. Though I should note: the gains weren't universal across tasks, and this approach requires additional inference compute. It's not a free lunch.
This connects to a broader trend I'm seeing in the VLA research space right now. Multiple groups are converging on the same basic insight: these models make fast, instinctive decisions that work well in common scenarios but fall apart when things get tricky. The question is what to do about it.
One approach, from the VLA-ATTC paper, introduces what they call a "cognitive clutch." The idea is to let the model run on autopilot most of the time, but when uncertainty spikes, switch into a deliberation mode where it generates multiple action candidates and picks the best one. They use a Relative Action Critic that compares actions in pairs rather than trying to estimate absolute values, which apparently makes the learning problem much easier. On the LIBERO-LONG benchmark, this reduces Pi0.5's failure rate by over 50%. That's a big claim, and I'd want to see more independent replication before getting too excited, but the direction makes sense.
Then there's AttenA+, which takes a completely different angle. Their argument is that robot trajectories are fundamentally heterogeneous, and treating all timesteps equally during training is a mistake. Low-velocity segments, where the robot is doing something precision-demanding like inserting a peg, matter more than high-velocity transit motions. So they reweight the training objective based on inverse velocity. It's architecture-agnostic, requires no extra parameters, and apparently pushes OpenVLA-OFT to 98.6% on the LIBERO benchmark. I find this kind of physics-aware training signal elegant, tbh. It's the sort of thing that seems obvious in retrospect.
ProgVLA goes further down the progress-tracking rabbit hole. They train auxiliary "progress heads" using offline RL objectives to give the policy an internal estimate of how far along it is in a task. This enables advantage-weighted and success-weighted imitation learning. What's notable here is the efficiency angle: a 0.1B parameter model is competitive with, and sometimes beats, much larger pretrained baselines. The gains concentrate on long-horizon and multi-object tasks, which makes sense. Those are exactly the scenarios where knowing your progress would help most.
All of this research is happening against a backdrop of increasingly sophisticated benchmarking. Colosseum V2 just dropped with 28 tasks across 13 categories, specifically designed to stress-test generalization. The results are, well, humbling. State-of-the-art methods including Pi0.5 show real limitations when you push them out of distribution. The researchers found strong correlations between simulation and real-world metrics, which is reassuring for the field. But the core message is that our current models are more brittle than their headline numbers suggest.
And then there's the memory question. RoboMME is a new benchmark focused specifically on long-horizon, history-dependent manipulation. Think tasks that require counting repeated actions or tracking objects that get temporarily occluded. They built 14 memory-augmented VLA variants on the Pi0.5 backbone to systematically explore different approaches. The finding that stuck with me: the effectiveness of memory representations is highly task-dependent. There's no single best approach. Each design has distinct advantages and limitations across different scenarios.
I think what we're seeing here is the field maturing past the "scale everything" phase into something more nuanced. Yes, bigger models and more data help. But there's clearly structure in robot manipulation that pure imitation learning doesn't fully capture. Success prediction, progress awareness, temporal criticality, memory, these all seem to matter, and they require different solutions.
Honestly, I'm not sure we have a unified theory yet for why frozen VLAs encode success information. The researchers speculate about it emerging from the structure of demonstration data, but it remains unclear whether this generalizes beyond the specific benchmarks tested. The probing results are compelling, but probing tells you what information is present, not necessarily what the model is using during normal operation.
What I do think is clear: the next generation of VLAs will probably look quite different from the current crop. Not necessarily bigger, but smarter about when to think hard versus act fast. More aware of their own uncertainty. Better at knowing where they are in a task. The foundations are already there in the representations. The question is how to unlock them without breaking everything else.
The push-plate success rate going from 26.7% to 44.3% isn't going to change the world. But it's a proof of concept that there's value hiding in these models that we haven't figured out how to extract yet. And in a field that sometimes feels stuck on incremental benchmark improvements, that's actually pretty exciting.
