Foundation Models Are Finally Learning to See the Way Robots Need To

The second paper is more ambitious. FOUND-IT, they're calling it. A system that builds hierarchical 3D scene graphs from a single uncalibrated monocular camera in real-time. According to the researchers, it runs on a Jetson Thor, which is impressive if true (those things are powerful but not exactly supercomputers).

What caught my attention is the "granularity on demand" aspect. The system adjusts how detailed its map is based on what task the robot is doing. During manipulation, it resolves small features like knobs on a stove. During navigation, it focuses on large objects. This is how humans actually work, right? You don't maintain a millimeter-accurate mental model of your entire house. You zoom in when you need to.

The really interesting bit: the task list isn't fixed. It adapts as the robot operates. For loco-manipulation (walking robots that also grab things), this could be significant. The authors claim 79% higher accuracy on some benchmark I'm not familiar with, so take that with appropriate salt.

Here's what I think is actually happening. These foundation models, the big pre-trained vision systems that have gotten so much attention in the AI world, are finally being adapted for robotics in ways that make practical sense. For years we had this disconnect where computer vision people were solving problems that looked impressive on benchmarks but didn't help robots navigate a cluttered shop floor.

The FOUND-IT team specifically mentions they added an extra head to an existing geometric foundation model (VGGT) to extract traversability information. That's clever engineering. You're not building from scratch, you're leveraging billions of dollars worth of training that someone else already did.

I called a former colleague at Siemens last week, asked what they're seeing in terms of vision system reliability on their logistics installations. He said it's "getting better, slowly." That tracks with my sense of things. We're not at the point where you can drop a mobile robot into an unknown warehouse and trust it to figure things out. But we might be getting closer.

The limitation I'd flag: both papers are from academic groups, and there's a long road from "works in the lab" to "works at 3 AM when the night shift supervisor accidentally leaves a pallet in the wrong spot." The FOUND-IT folks tested their system on YouTube apartment tour videos, which is a nice robustness check but not exactly an industrial environment.

Still. Progress is progress. And after watching vision systems be the bottleneck for most of my career, I'll take cautious optimism over the alternative.

Sarah Williams · 19 Jun · 3 min