The "robotics generalist" question gets a useful new benchmark

Robotics has had a lot of benchmarks, and the field has known for years that most of them overstate how much actual generalisation models are capable of. A new benchmark suite, RoboGen, addresses the criticism directly.

The arXiv paper proposing RoboGen, and an IEEE Spectrum analysis, describe a benchmark designed specifically to evaluate cross-task and cross-environment generalisation.

What is different

Most existing manipulation benchmarks evaluate model performance on tasks that are minor variants of the training conditions. New object colours, slightly different starting positions, different table heights. Models perform well, and the field celebrates "generalisation".

The criticism, fairly, has been that this is not generalisation in any meaningful sense. It is interpolation across small variations within a distribution the model already saw.

RoboGen is structured to test something harder. The benchmark holds out entire task categories from training. It includes environments with object configurations the model could not have encountered during training. It evaluates on robot platforms with kinematic structures different from those used in training.

The early results from the original paper are bracing. State-of-the-art models that score in the 80s on conventional benchmarks score in the 30s and 40s on RoboGen. The gap reflects how much current "generalisation" is interpolation within a known distribution.