The VLA Metrics Problem: Why Your Robot's Best Checkpoint Might Be Its Worst

The culprit? Easy-to-predict joints masking hard-to-predict ones in the aggregate. The mobile base converged slowest in SmolVLA and became the limiting factor. In the expert-only fine-tuning of the larger model, total MSE dropped below baseline but arm accuracy actually degraded. The arm is, you know, sort of important for a manipulation task.

The researchers' conclusion is straightforward: per-group error is a more reliable signal than total MSE for checkpoint selection on robots with heterogeneous action spaces. Call me old-fashioned, but this feels like something we should have been doing from the start.

When all your options are bad

The second paper tackles a different but related problem. A lot of recent VLA work uses what's called test-time scaling, you sample K candidate action chunks at inference and execute whichever one your verifier thinks is best. Methods like RoboMonkey, SEAL, MG-Select, and V-GPS all do some version of this.

Here's the thing nobody was talking about: what happens when all K candidates are unsafe?

The answer, it turns out, is that the system just picks one and executes it anyway. No warning, no abstention, no nothing. The BOKBO paper (that stands for "Best of K Bad Options," which I appreciate for its honesty) proposes the first conformal abstention layer for this scenario.

But the really damning part isn't the solution, it's what they found when they dug into why existing approaches fail. The policy-internal confidence scores that these systems use to judge action quality? Under perturbation-based K-sampling, they correlate at 0.98 with the action-noise hyperparameter σ. Their correlation with actual safety violations? At the noise floor. Basically zero.

Let me say that again because it's important: the confidence proxy these systems use to pick actions correlates almost perfectly with a tuning parameter and basically not at all with whether the action is actually safe.

The researchers also identified what they call a "methodological pitfall" in the field. Force thresholds for detecting unsafe behavior were being set globally, well below typical expert manipulation forces. This conflated normal manipulation with unsafe behavior and inflated violation rates by 5x. That's not a rounding error, that's the difference between a paper that looks good and one that doesn't.

The déjà vu is strong

If you covered autonomous vehicles in the 2015-2018 period, all of this should sound familiar. We went through years of debates about disengagement metrics, about whether miles-between-interventions was meaningful, about companies cherry-picking evaluation scenarios. The fundamental issue was always the same: aggregate metrics that looked good on paper but masked the failure modes that mattered.

The VLA community is, in some ways, repeating those mistakes. Total MSE across heterogeneous joints. Confidence scores that track hyperparameters instead of outcomes. Safety thresholds set without reference to what normal operation looks like. These aren't subtle problems! They're the kind of thing that seems obvious in hindsight but somehow persists because everyone's using the same benchmarks and nobody wants to be the first to say the emperor has no clothes.

I should be fair here. Both papers propose solutions, and they're sensible ones. Per-group error tracking is straightforward to implement. BOKBO's conformal abstention layer provides what the authors call "finite-sample distribution-free guarantees" on violation rates. The Mondrian variant raised the minimum per-task conditional hold fraction from 0.71 to 0.93 in their experiments. That's real progress.

But what do I know. Maybe the kids building these systems have already figured this out and these papers are just catching up to industry practice. I'd love to hear from people actually deploying VLA models at scale about whether this matches their experience (my email's on the about page).

The coverage question

The BOKBO paper reports some specific numbers worth noting. At ε = 0.05 on their libero_object benchmark with OpenVLA-OFT, they achieved 78% coverage and 70% net task success. The conditional CRC bound held on 86% of bootstrap splits. Results were stable across 5 training seeds and replicated on libero_spatial as a secondary benchmark.

Those numbers are, well, they're not great? 70% net task success means 30% failure, and while having calibrated uncertainty about when you'll fail is valuable, it's not the same as not failing. The authors are upfront about this being a first step, which I appreciate.

The token-level temperature sampling results are interesting too. The correlation failure they identified was "mechanism-specific and partially mitigated" under policy-stochasticity-based sampling rather than perturbation-based. This suggests the problem isn't inherent to K-sampling approaches, just to how most people are currently implementing them.

What this means going forward

I think we're at an inflection point with VLA models. The hype is real, the demos are impressive, and the investment money is flowing. But the evaluation infrastructure hasn't caught up to the ambition. We're still using metrics designed for simpler systems on robots that are genuinely complex.

The per-group error paper's code is available on GitHub. The BOKBO paper provides detailed methodology. Neither is particularly hard to implement. The question is whether the field will adopt these approaches or whether we'll spend another few years publishing impressive aggregate numbers while the robots keep failing in production.

I'm cautiously optimistic, actually. The AV industry eventually developed better evaluation standards, even if it took longer than it should have. The robotics community has the advantage of learning from that history. Whether they will remains unclear.

One thing I'm less certain about is how these findings generalize beyond the specific platforms tested. The Toyota HSR is a particular kind of robot with a particular action space. Mobile manipulators with wheeled bases might have different failure modes than, say, humanoids or fixed-base arms. The BOKBO experiments were on LIBERO benchmarks, which are useful but not exhaustive. More replication across platforms would help.

For now, if you're training VLA models on robots with multiple joint groups, maybe don't trust your aggregate MSE. And if you're using K-sample inference with a verifier, maybe check whether your confidence scores actually correlate with the thing you care about. These seem like reasonable asks.

The field is moving fast, which is exciting. But moving fast in the wrong direction just gets you lost faster. I've seen enough hype cycles to know that the unsexy work of measurement and evaluation is usually what separates the technologies that actually work from the ones that just demo well.

These two papers are doing that unsexy work. More of this, please.

Mark Kowalski · 9 hours ago · 6 min