New Benchmarks Expose a Hard Truth: Vision-Language Models Can't Keep Robots Safe Yet

The collision grounding problem

The second paper, TouchSafeBench, tackles something even more basic: can vision-language models tell if a robot is about to hit something?

The researchers built a physics-grounded benchmark with 2,940 simulated indoor episodes, testing models on two seemingly simple tasks. First, classify whether the current state is safe. Second, warn about imminent collision before contact actually happens.

The best average Macro-F1 score across tested models stayed below 50%. For context, random guessing on a binary classification would get you 50%. These are frontier models, and they're performing at or below chance on determining whether a robot is about to collide with a person.

One finding stood out to me: "explicit depth is not automatically transformed into robot-body collision evidence." The models can perceive depth information. They just can't reliably connect that perception to the physical reality of where the robot's body actually is in space. Robot-scene contact proved consistently harder to detect than human-contact risk, which suggests the models haven't internalized basic geometric reasoning about the robot itself.

Why this matters for deployment

There's a temptation in this industry to treat vision-language models as a shortcut to general-purpose robot intelligence. Feed in camera data, add some language instructions, and the model figures out the rest. These papers suggest that approach has fundamental limitations.

The TouchSafeBench authors put it bluntly: "visual fluency does not imply physical accountability." A model can describe what it sees in perfect prose while remaining completely unreliable as a safety monitor. That's an ambitious gap to close, and it's unclear whether scaling current architectures will solve it.

From my time building hardware, I learned that safety systems need to be boring. They need to work the same way every time, with failure modes you can characterize and bound. The current generation of VLMs, for all their impressive capabilities, remain too unpredictable for safety-critical applications.

Both papers point toward similar requirements: explicit geometric reasoning, proper grounding of the robot's own body in space, and coordination mechanisms that go beyond text-based cues. Whether those capabilities emerge from larger models or require architectural innovation remains to be seen.

The code for CARLA-Air is available on GitHub. TouchSafeBench will be released upon paper acceptance. For teams working on multi-robot coordination or human-robot collaboration, these benchmarks offer a sobering reality check, and a concrete starting point for measuring progress.

James Chen · 6 hours ago · 4 min