New Benchmarks Expose How Badly Vision-Language Robots Fail at Retail Tasks

A second paper tackles the "how" of robot interaction, not just the "where." Released alongside RoboBenchMart in the broader research community, AFUN takes a different angle on the same underlying problem: how do you build robot AI that generalizes to environments it hasn't seen?

AFUN, which the authors position as a step toward an "affordance foundation model," attempts to predict both where a robot should interact with an object and how it should move after making contact. From a single RGB-D image and a language description, the system outputs a functional mask (the region to touch) and a 3D post-contact motion curve (what to do next).

The results are, I'll admit, impressive on paper. AFUN outperforms all baselines across eight test sets from four benchmarks, improving mean gIoU by 23.9 points and cIoU by 26.3 points. For contact-point prediction, it achieves a 12.7 to 61.3 percent hit-rate improvement over the best competing method. The researchers also claim it can be deployed for real-world manipulation without fine-tuning for specific robot hardware.

That last claim is the one I'd want to see proven at scale. From my time building hardware, I learned that "works in the lab" and "works in production" are separated by a canyon of edge cases. The AFUN team built a large-scale data pipeline that converts heterogeneous sources (robot data, human demonstrations, simulations, real-world scans) into a shared format. That's genuinely useful infrastructure. Whether it's enough to bridge the sim-to-real gap remains unclear.

What these papers share is a diagnosis: current generalist robots aren't general enough. RoboBenchMart shows that VLA models trained on household tasks don't transfer to retail. AFUN shows that predicting affordances (what you can do with an object) requires both spatial understanding and motion prediction, and most existing methods only do one or the other.

Neither paper claims to have solved the problem. RoboBenchMart explicitly frames itself as a benchmark, not a solution. AFUN calls itself "a step towards" a foundation model, not the foundation model itself. That kind of honesty is refreshing.

The real test is what happens next. For retail automation specifically, the path forward probably involves a lot more domain-specific training data. RoboBenchMart's procedural generator could help, but we're still talking about simulated environments. The jump to physical robots in physical stores, with real products that dent and spill and get restocked in random orientations, that's an engineering problem that benchmarks alone won't solve.

For robotics investors and operators, the takeaway is simple: be skeptical of demos. A robot that performs well on standard benchmarks may perform poorly in your specific environment. Ask vendors which benchmarks they've tested against. Ask whether those benchmarks resemble your actual use case. If they can't answer, that's an ambitious gap in their story.

Both papers are open-source, which means anyone can run the evaluations. RoboBenchMart includes fine-tuned baseline models. AFUN provides a project page with code and data. The tools exist now to hold robot AI claims to a higher standard.

Whether the industry uses them is another question.

Sarah Williams · 4 hours ago · 5 min