Robot Arms That Can't Follow Orders in French: The Language Problem Nobody's Talking About
A new study finds that AI-driven robot systems trained in English fall apart when you give them instructions in any other language. For global factory floors, that's a real problem.
By
Most of the coverage I've seen on Vision-Language-Action models, the AI systems that let robots understand spoken or written instructions and act on them, has been breathless. New benchmark, new demo video, impressive numbers. What it mostly skips over is the part where you take that robot off the English-language test bench and put it on an actual factory floor in, say, Guadalajara or Stuttgart.
I'll be honest, when I saw the headline from arXiv about a "multilingual gap" in VLA models, my first reaction was: of course there is. When I was at Kuka, we spent months on localization problems that had nothing to do with AI, just getting control software to handle German decimal formatting correctly so a machine didn't think 1.500 meant fifteen hundred millimetres instead of one and a half. Language and locale have always been an afterthought in industrial systems. Apparently that habit carried over into the neural network era.
The paper, out of a research group studying robotics and language models, is the first systematic look at whether VLA models can follow instructions in languages other than English. Short answer: they mostly can't, or at least not well. Even when the underlying language model powering the system has multilingual capability baked in, that capability doesn't seem to survive the training process that turns the language model into a robot controller. The researchers found substantial performance drops across multiple languages, and the degradation correlated with both how well the model understood the instruction and how well it executed the resulting action. Both ends of the pipeline take a hit.
Related coverage
More in Industrial
Three new papers on offline-to-online reinforcement learning suggest robots are getting much better at picking up skills without starting from scratch every time.
Robert "Bob" Macintosh · 6 hours ago · 4 min
A pair of arxiv papers on robot planning caught my eye this week. One's about object-aware decision-making, the other about robots refining their own plans mid-thought. Both point in the same direction.
Robert "Bob" Macintosh · 9 hours ago · 4 min
Four new papers on robot manipulation landed this week, and honestly, a couple of them are the real deal.
Robert "Bob" Macintosh · 9 hours ago · 5 min