The Sim-to-Real Gap Is Finally Closing, and Nobody's Celebrating

Is this the final answer? Remains unclear. But the approach of tiered simulation fidelity, fast and rough for learning, slow and accurate for validation, feels like it might stick around.

Learning to see what you've already seen

The second paper that caught my attention is called MAD, which stands for Mapping-Aware Dreamer, and it comes from researchers working on agile quadrotor flight (full paper here). The insight is deceptively simple: instead of training a drone to react to what it currently sees, train it to remember what it's already seen and reason about what that means for where obstacles might be.

This is the kind of thing that sounds obvious when you say it out loud but turns out to be genuinely hard to implement. The drone maintains internal "occupancy and visibility grid maps" in its latent state, which is a fancy way of saying it builds a mental model of the space around it, including the parts it can't currently see because they're behind it or temporarily occluded.

The results are impressive in that understated way academic papers have of burying the lede. They flew a real quadrotor through a forest at 5.05 meters per second using a consumer-grade Intel RealSense depth camera. In simulation they hit 9.66 m/s. For context, that's fast enough to seriously hurt someone if something goes wrong, and the fact that it didn't go wrong suggests the sim-to-real transfer is working better than I would've expected.

The paper also mentions they're releasing the source code, which, if you're a grad student working on this stuff, is probably more valuable than the paper itself.

The perching problem

The third paper is narrower in scope but weirdly compelling. It's about getting drones to land on moving, tilted platforms (arXiv link). Think: drone landing on a boat deck in rough seas, or on a moving truck, or on another larger drone. The researchers call it PerchRL.

The challenge here is that the drone's camera can only see so much. When you're trying to land on a platform that's moving unpredictably and also tilted at an angle, you're going to lose visual contact repeatedly. The platform moves out of your field of view, you have to guess where it went, and then reacquire it.

Their solution involves what they call "visibility-aware state augmentation" and "active perception rewards," which basically means the drone gets rewarded for maneuvering to keep the landing target in view, not just for getting closer to it. It's learning to look where it needs to look, which sounds like something that should be trivial but apparently isn't.

I don't have access to the actual success rates from their real-world tests (the paper mentions "extensive" experiments but the specific numbers weren't in the abstract), so I'm hedging here. But the fact that they claim successful deployment across multiple different quadrotor platforms suggests the approach generalizes, which is often where these things fall apart.

So what does this actually mean

Here's where I'm supposed to tell you this changes everything and the future is now and we're all going to have drone swarms delivering our packages by Christmas. I'm not going to do that, because I've seen this movie before.

What I will say is this: the infrastructure for training robot behaviors is getting dramatically better, and the gap between what works in simulation and what works in reality is shrinking. Five years ago, a paper claiming 30,000x speedup over standard simulation would've been dismissed as either wrong or useless (too fast usually means too unrealistic). Now it's being validated against high-fidelity simulators and then deployed on real hardware.

The policy implications are, well, we don't really have policy implications yet because the policymakers are still trying to figure out what a large language model is. But autonomous drones that can navigate cluttered environments at high speed, land on moving platforms, and coordinate in groups for tracking missions? That's military tech. That's surveillance tech. That's also search-and-rescue tech and environmental monitoring tech, but let's not pretend the funding for this research is coming from humanitarian concerns.

I'm not saying we should panic. I'm saying the kids building this stuff are solving the hard problems faster than the regulatory frameworks can keep up, which is, in a way, the story of every technology I've covered since the 90s. The difference is that this technology flies and makes decisions on its own.

If you want to argue with me about any of this, my email's on the about page. I still check it, unlike certain younger colleagues who shall remain nameless.

Mark Kowalski · 3 days ago · 5 min