The Real Problem With Teaching Robots: We Keep Pretending Humans Aren't Part of the Loop

What struck me reading this paper is how much effort goes into handling what they call "perception uncertainties." Translation: the robot doesn't actually know exactly where things are. This is a problem that's been with us since the beginning of robotics, and it's refreshing to see researchers actually testing against it rather than assuming perfect sensor data.

Meanwhile, back on Earth, a team from Tsinghua University has been wrestling with a different version of the same fundamental tension. Their paper on OHP-RL (Online Human Preference as Guidance in Reinforcement Learning) tackles something that's been bugging me for years about human-in-the-loop robotics: we keep treating human interventions like they're just better versions of robot actions, when really they're something else entirely.

Here's the insight, and it's a good one: when a human grabs the joystick and takes over from a robot, they're not necessarily demonstrating the optimal action. They're expressing a preference. They're saying "not that, something more like this." The difference matters! If you train a robot to exactly imitate human interventions, you're assuming humans are perfect demonstrators. They're not. (I've been driving for 40 years and my parallel parking is still, well, let's move on.)

The OHP-RL framework introduces what they call a "preference gate" that decides when and how much human input should influence the policy. It's sort of like having a dial between full autonomy and full human control, except the dial adjusts itself based on the situation. The results on contact-rich manipulation tasks (think: inserting pegs, pressing buttons, that kind of thing) show faster learning and, crucially, less human effort over time.

The pattern I keep seeing across both these papers, and honestly across most of the robot learning work I've read in the past five years, is a gradual retreat from the dream of pure end-to-end learning. Remember when everyone was convinced you could just throw a neural network at raw sensor data and out would pop a working robot? The kids in the labs back then were so sure they'd figured it out.

What we're seeing now is more... realistic. The space manipulator paper uses prior policies as guidance. The human-preference paper explicitly builds in mechanisms for human oversight. Both papers spend considerable effort on robustness testing, on what happens when things go wrong, which tells me the researchers have actually tried to deploy these systems outside simulation.

But what do I know. I'm just a guy who's watched three generations of "this time it's different" claims in AI and robotics. Maybe the fourth time's the charm!

The uncomfortable question neither paper really answers is this: how do you scale this stuff? The space manipulator work was tested in simulation (they've released the code on GitHub, which I appreciate). The human-preference work was tested on a single Franka robot arm with what appears to be a dedicated human operator providing feedback. Both are legitimate research contributions, but neither tells us much about what happens when you try to deploy these approaches across hundreds or thousands of robots.

This is the self-driving car hype cycle all over again. The technology works great in controlled conditions with plenty of human oversight. The question is whether it works when you remove the training wheels, and we don't know yet. The papers are honest about their limitations (the space manipulator paper specifically notes they tested under "various challenging scenarios" but simulation is still simulation), but the press releases and Twitter threads that inevitably follow tend to sand off those edges.

What actually matters here is the convergence I'm seeing across different subfields. Space robotics and tabletop manipulation don't have much in common operationally, but both teams independently arrived at similar conclusions: pure autonomy isn't the goal, coordinated human-robot systems are. The space manipulator work coordinates two control agents (arm and base). The human-preference work coordinates human judgment with robot learning. Both are moving away from the "robot does everything alone" paradigm that dominated the field's imagination for decades.

Is this progress? Absolutely. The technical contributions in both papers are solid. The DACMP framework's handling of dynamic coupling is genuinely clever, and the preference-gate mechanism in OHP-RL addresses a real gap in how we think about human feedback. But I'd feel better about the state of the field if more papers were honest about how far we are from the science fiction version of robotics that keeps getting promised in funding proposals.

The space manipulator paper reports task success rates but doesn't define what counts as success in terms that a spacecraft operator would recognize. The human-preference paper shows lower intervention effort but doesn't quantify what "lower" means in absolute terms (is it one intervention per task? Ten? We're not told). These are the details that matter for deployment, and they're consistently the details that get glossed over.

For practitioners, both papers offer useful technical contributions. The TESG mechanism for switching between learned and prior policies could be adapted to other domains where you have imperfect expert knowledge. The preference-gate approach could help with any human-in-the-loop system where you want to reduce operator fatigue over time. The code availability for the space manipulator work is particularly helpful.

For everyone else, the takeaway is simpler: robots are getting better at working with humans, not at replacing them. Whether that's what the funding agencies want to hear remains unclear, but it's what the research actually shows. If you want to argue about it, my email's on the about page.

Mark Kowalski · 3 hours ago · 5 min