Reinforcement Learning Is Finally Making Drones Fly Better, and I've Seen This Before

This is a hybrid approach, and honestly it makes a lot of sense. You get the adaptability of reinforcement learning for high-level decisions while keeping the battle-tested reliability of PID for the stuff that needs to happen in milliseconds. The researchers used Soft Actor-Critic (SAC), which is a model-free algorithm that works well with continuous action spaces. Training was faster than conventional RPM controllers, and the path-following was smoother and more accurate.

Now here's where I have to pump the brakes a little, because someone has to.

We don't actually know how these controllers perform over thousands of flight hours. We don't know how they handle sensor degradation, motor wear, or the kind of weird edge cases that only show up when you've deployed ten thousand units. The papers show promising results under controlled conditions, which is necessary but not sufficient for anyone thinking about putting this stuff in production.

I've talked to enough drone companies over the years to know that the gap between "works in the lab" and "works at scale" is where most promising technologies go to die. Not because they're bad ideas, but because the real world is full of surprises that no training environment can fully capture. One engineer at a delivery drone startup told me (this was maybe 2019) that their biggest problem wasn't the algorithms, it was the fact that birds kept attacking their drones and no amount of simulation prepared them for that.

What's actually new here, and I want to be fair about this, is the maturity of the RL techniques being applied. TD3 and SAC are both relatively recent algorithms that address some of the instability problems that plagued earlier deep RL approaches. The fact that researchers are getting good results with off-the-shelf algorithms rather than needing custom architectures suggests we might be approaching something like commodity RL control. Maybe.

The hybrid approach in the second paper is particularly interesting because it suggests a path forward that doesn't require throwing out everything we know about classical control. You don't need to trust the RL agent with everything, just the parts where adaptability matters most. This is the kind of pragmatic engineering that actually ships products, as opposed to the "pure RL will solve everything" attitude that gave us a lot of impressive demos and not much else in the 2010s.

But what do I know. I've been covering tech long enough to have been wrong about plenty of things. I thought smartphones would never replace laptops for real work (partially wrong), I thought autonomous vehicles would be everywhere by 2020 (very wrong), and I thought robotics would stay a niche industrial thing forever (wrong in an interesting way).

The question I keep coming back to is whether these RL controllers can handle the certification process. Aviation regulators, even for small drones, want to understand why a system makes the decisions it makes. Neural networks are famously opaque. The hybrid approach might actually help here, since you can point to the PID layer and say "this part we understand completely, the RL just tells it what to aim for." It's not a complete answer but it's something.

For the kids building drone startups right now, and I mean that affectionately, these papers are worth reading carefully. Not because they're going to revolutionize your flight stack tomorrow, but because they show where the research community thinks the field is heading. The emphasis on real-world testing, the hybrid architectures, the use of mature RL algorithms, this all points toward a more practical era of learning-based control.

Whether that era actually arrives or whether we're in for another round of overpromising and underdelivering remains unclear. I've learned to wait for the second and third papers that try to replicate results before getting too excited. Science moves slower than press releases suggest, and that's probably a good thing when we're talking about machines that fly.

The simulation results look good. The hardware tests look promising. But I'll believe we've cracked RL-based flight control when I see these systems flying commercial routes for a year without incident. Until then, it's interesting research, nothing more and nothing less.

If you want to argue about any of this, my email's on the about page. I still check it, unlike some people.

Robert "Bob" Macintosh · 18 Jun · 4 min