画像クレジット: Lottie animation by Centre Robotics (LottieFiles Free, used with credit). · source
Why do we keep learning the same lesson over and over?
I've been covering tech long enough to remember when neural networks were going to replace programmers entirely (they didn't), when expert systems would automate doctors (nope), and when self-driving cars were five years away (that was fifteen years ago). The pattern is always the same: researchers announce that AI can do X autonomously, then quietly discover that actually, humans need to stay involved, and the real breakthrough is figuring out exactly how much involvement and when.
Two papers crossed my desk this week that, on the surface, have nothing to do with each other. One's about space manipulators, the kind of robotic arms that service satellites. The other's about teaching a Franka robot arm to do contact-rich manipulation tasks in a lab. Different domains, different teams, different continents probably. But they're both grappling with the same fundamental problem, and arriving at remarkably similar conclusions.
Researchers from (I'm guessing) a Chinese university have been working on what they call "Dual-Agent Coordinated Manipulation Planning," or DACMP. The setup: you've got a spacecraft with a 6-degree-of-freedom robotic arm attached to it. You want the arm to reach out and grab something, maybe a satellite that needs servicing, maybe debris that needs clearing. Simple enough on Earth. In space? The moment that arm moves, Newton's third law kicks in and your entire spacecraft starts rotating the other direction.
This is the dynamic coupling problem, and it's been a headache for space robotics since the Canadarm days. The paper's contribution isn't the problem statement, it's the solution architecture. They use deep reinforcement learning, which is standard these days, but with a twist they call "Timestep-level Expert Switching Guidance" or TESG.
関連記事
More in Research
Three papers crossed my desk this week that suggest we're finally getting serious about making robots do what we actually tell them to do.
Robert "Bob" Macintosh · 25 mins ago · 4 min
Researchers are finding ways to train robots with far less data, using human corrections and physics simulators instead of millions of demonstrations.
James Chen · 25 mins ago · 6 min
A batch of new research papers suggests we might finally be solving the sample efficiency problem that's plagued robotics for years, and I've seen this inflection point before.
Mark Kowalski · 25 mins ago · 5 min
Two new papers show hexapods and transformable drones doing whole-body manipulation, which is the kind of unsexy problem that actually matters.
What TESG does, basically, is let a prior policy (think of it as a rough expert demonstration) guide the learning agent at specific moments during training. Not constantly, not randomly, but at particular timesteps where the expert's knowledge is most valuable. The system learns when to listen and when to explore on its own.
The results are pretty good! Higher task success rates than baseline DRL algorithms, better control precision, and the thing actually works under system constraints, environmental disturbances, and perception uncertainties. That last bit matters because space is full of surprises and you can't exactly run back to fix your robot if it gets confused.
Meanwhile, back on Earth, a separate team has been working on what they call OHP-RL, which stands for "Online Human Preference as Guidance in Reinforcement Learning." They're dealing with a Franka robot doing contact-rich manipulation, the kind of tasks where the robot has to push, slide, or otherwise interact with objects in ways that are hard to simulate perfectly.
Their insight is that when humans intervene to correct a robot, they're not just saying "do this exact action." They're expressing a preference, a relative judgment that this behavior is better than that behavior under these conditions. Previous approaches treated human interventions as demonstrations to imitate. OHP-RL treats them as preference signals that should shape learning differently depending on the state.
They introduce something called a "state-dependent preference gate" (I know, the names in academic papers are something else) that decides when and how much human feedback should influence the policy. The robot can still explore on its own, still learn from trial and error, but it has guardrails that activate when it's about to do something the human would object to.
Three real-world tasks, strong success rates, faster convergence, and here's the kicker: substantially lower human intervention effort than prior approaches. The robot needed less hand-holding to learn better behavior.
Look, I've seen this movie before. The self-driving car industry spent a decade trying to remove humans from the loop entirely, then quietly pivoted to "supervised autonomy" and "human-machine teaming" and whatever other euphemism means "actually we need a person watching." The difference between those efforts and what these papers describe is that these researchers are being honest about it from the start.
Both papers are essentially arguing the same thing: pure autonomous learning is inefficient and potentially unsafe, human guidance is valuable but expensive and imperfect, so the real engineering challenge is building systems that can extract maximum value from minimal human input at exactly the right moments.
This is not a failure of AI! This is, if anything, a maturation of the field. The kids coming up now (and I mean that affectionately, some of them are brilliant) seem to understand something that earlier generations of researchers sometimes missed: the goal isn't to replace humans, it's to build systems that work well with humans.
The TESG mechanism in the space paper and the preference gate in the OHP-RL paper are doing conceptually similar things. They're both asking "when should the system defer to prior knowledge or human judgment, and when should it trust its own learning?" That's a much more interesting question than "how do we remove humans entirely," and it's a question that generalizes across domains.
I don't want to oversell this. These are academic papers, not deployed systems. The space manipulator work is validated in simulation (they've released code on GitHub, which is good), and the Franka experiments are real but still laboratory conditions. The gap between "works in the lab" and "works in production" remains, as always, substantial.
But the direction feels right to me. Call me old-fashioned, but I've always been skeptical of approaches that assume we can engineer our way out of needing human judgment. The messy reality is that robots operating in unstructured environments will encounter situations their training didn't anticipate, and when that happens, you want a system that knows how to ask for help rather than one that confidently does the wrong thing.
The space application is particularly interesting because the stakes are so high and the feedback loop is so slow. You can't exactly teleop a robot arm from Earth when there's a multi-second signal delay. The system has to be autonomous enough to act, but constrained enough to act safely. DACMP's approach of baking in prior policy guidance at the training level, rather than relying on real-time human oversight, seems like a reasonable way to thread that needle.
For the terrestrial manipulation work, the implications are more immediate. Contact-rich tasks are exactly the kind of thing that current industrial robots struggle with, and exactly the kind of thing that would unlock new applications in manufacturing, logistics, and eventually home robotics. If OHP-RL's approach to human-in-the-loop learning actually reduces the burden on human trainers while improving outcomes, that's a meaningful step toward practical deployment.
Will either of these specific techniques become standard? Too early to say, and I only found these two papers on this particular theme this week, so it's possible I'm pattern-matching on limited data. But the underlying philosophy, that autonomous systems should be designed to incorporate human guidance efficiently rather than to exclude it entirely, that feels like where the field is heading.