VLA models are succeeding at tasks while failing at safety, and we're only now measuring it

Why this happens (I think)

I initially thought this was just a training data problem. Train on demonstrations that don't emphasize safety, get a model that doesn't prioritize safety. Simple.

But after reading the second paper, I'm less sure it's that straightforward.

PaCo-VLA takes a different approach entirely. Instead of trying to make the VLA model itself safer, it treats VLA outputs as "compliance proposals" rather than direct motor commands. There's a high-frequency passivity shield that governs these proposals through energy accounting and boundary checks. The VLA suggests what to do; a separate system decides whether that's actually safe to execute.

The framing here is interesting. The researchers argue that VLA models, by design, output at low rates and lack the reliability needed for direct control in force-sensitive tasks. You might be wondering why we'd give them direct control in the first place. Tbh, I think the answer is that it's easier, and until now we weren't measuring the consequences.

PaCo-VLA achieved zero passivity violations in their connector-insertion experiments, even under what they call "adversarial compliance shifts." That's a specific claim about a specific task, and I should note this is based on their own evaluation (I couldn't find independent replication). But the architecture itself, decoupling semantic reasoning from contact physics, seems like it addresses something fundamental.

What we still don't know

A few things remain unclear to me.

First, it's too early to say whether the SafeVLA-Bench metrics will become standard. The framework uses Signal Temporal Logic specifications, which is elegant but adds complexity. Will other researchers adopt it, or will we keep publishing success rates because they're simpler to compute?

Second, the PaCo-VLA approach works for contact-rich manipulation, but I'm not sure how it generalizes. Connector insertion is a well-defined task with clear force constraints. What about messier manipulation scenarios where "safe" is harder to specify?

Third (and this is something I should know better, but I'll admit I don't): we don't have great data on what violation rates would be acceptable for real-world deployment. Is 13 percent unsafe too high? Almost certainly. Is 1 percent? I genuinely don't know how to think about that threshold.

So what

Here's my takeaway, and I realize this might sound pessimistic.

We've been optimizing VLA models for the wrong metric. Or rather, we've been optimizing for an incomplete metric. Task success is necessary but not sufficient. A robot that completes 95 percent of tasks while applying excessive force in 40 percent of them isn't ready for deployment. It's ready for more research.

The good news is that both papers suggest paths forward. SafeVLA-Bench gives us tools to actually measure the problem. PaCo-VLA demonstrates that architectural choices can address it, at least in constrained scenarios.

The less good news is that a lot of existing VLA research might need to be reevaluated. Those impressive success rate numbers we've been celebrating? Some unknown fraction of them involve robots succeeding in ways we wouldn't actually want.

I think we're going to see safety-aware benchmarking become standard over the next year or two. The alternative, deploying models that complete tasks unsafely, isn't really viable once you move beyond simulation. But that transition will probably be messier than anyone wants to admit.

For now, when you see a VLA paper touting high success rates, it's worth asking: success at what cost?

Sarah Williams · 2 hours ago · 7 min