Foundation Models Are Getting Smarter About What They Don't Know

The second paper, on keypoint imitation learning, is more my speed because they actually ran real robots. Over 2000 real-world rollouts, which is the kind of validation you almost never see. The approach uses visual foundation models to extract keypoints from a single demonstration, then uses those as an intermediate representation for learning.

Here's the thing that caught my eye: they're honest about the limitations. KIL hit 75% success across five tasks, which beat the RGB baseline at 47%, but only matched an alternative method called S2-diffusion at 73%. They're not claiming they've cracked the problem. They explicitly say their approach inherits the limitations of whatever foundation model you use for keypoint extraction. That kind of honesty is, well, refreshing.

What both papers share is a pragmatic approach to using these big foundation models. Nobody's claiming you can just drop SAM or DINO into a robot and have it work. The question is how you adapt them efficiently, and both teams found ways to do that without needing massive datasets or compute budgets.

I called my old colleague at Siemens last week about something unrelated, and we ended up talking about this exact problem. His team is apparently spending a lot of time on domain adaptation for their inspection systems. The conventional wisdom has been that you need thousands of labeled images from your specific environment to make these models work. If RepSAM's approach holds up, that changes the math significantly.

Now, I should be clear about what we don't know yet. Both papers are from research labs. RepSAM was tested on benchmarks and controlled manipulation tasks, not a 24/7 production line with dust and vibration and operators who bump into things. The keypoint paper acknowledges that their foundation models struggle with certain failure modes. It's too early to say whether these techniques will survive contact with actual manufacturing environments.

But the direction is right. When I started in this industry, getting a vision system to work meant months of hand-tuning parameters for every new part. Then we got deep learning, which helped but required huge datasets. Now we're seeing methods that might actually let you adapt a general-purpose model to your specific application without a research team and a GPU cluster. That's the kind of progress worth paying attention to.

James Chen · 25 Jun · 5 min