The Teleoperation Problem Might Be Solving Itself, and That Should Worry Us

You might be wondering why this matters if you still need some human involvement. The answer is reset time. Anyone who's watched robot training knows the brutal inefficiency: move object, robot attempts task, fails, human resets object, repeat. Eliminating physical props during demonstration capture could dramatically speed up data collection.

But here's where I start getting uncertain. A third paper, RDGen, takes this further by using reinforcement learning policies as trajectory generators rather than final controllers. They train in simulation, transfer to real robots, harvest successful rollouts as demonstrations, then use those to train VLAs. The paper claims these RL-generated demonstrations produce "significantly smoother trajectories" than human teleoperation and yield better downstream performance.

I should know this better, but I'm not entirely clear on why smoother necessarily means better. Human demonstrations are messy because human motion is messy. Real-world deployment involves messy situations. There's an argument (one I find somewhat compelling, tbh) that training on overly clean data could hurt generalization.

The RDGen team addresses this partially, they're not claiming RL replaces humans entirely, just that it provides "more reliable and consistent supervisory signals." But the trajectory is clear: less human involvement in training data, more synthetic generation.

A survey paper published this week on arXiv frames the broader picture well. It categorizes approaches for deriving robot training data from human videos into four classes: latent action representations, predictive world models, explicit 2D supervision, and explicit 3D reconstruction. The authors highlight three open challenges, and the one that stuck with me is "grounding video-derived supervision into robot-executable actions under embodiment and viewpoint heterogeneity."

Translation: we're getting better at extracting knowledge from videos of humans doing things, but mapping that to robots with different bodies and perspectives remains hard. The AGILE framework attempts this through what they call "agentic generation," using vision-language models to guide synthesis of complete object meshes from occluded video footage. It's clever, but it's also another step toward removing humans from the loop.

So what's my actual concern? It's not that synthetic training data is bad. The LEGS results suggest it can be genuinely effective. It's that we're optimizing for scale and cost reduction without fully understanding what we lose.

Human teleoperation demonstrations capture something beyond just trajectories. They encode human judgment about what matters, where to be careful, when to pause. The HOIST paper on suspended load manipulation (think: humanoids carrying swinging payloads) explicitly notes that imitation learning provides "safe initial behavior." That safety comes from human intuition about physics, about failure modes, about caution.

When we replace human demos with procedurally generated motion primitives or RL rollouts, we might be optimizing for task success while losing something harder to measure. I don't have evidence this is happening, it's too early to say, but I also don't see anyone systematically checking.

The economics are going to push this forward regardless. Teleoperation is expensive, probably $50-100 per hour of demonstration time when you factor in operator training, equipment, and iteration. Synthetic generation could drop that by an order of magnitude or more. For companies trying to scale humanoid deployment, that math is irresistible.

I think we're going to look back at 2025 as the year the teleoperation bottleneck started breaking. Whether that's good news depends on questions we haven't figured out how to ask yet.

Robert "Bob" Macintosh · 5 hours ago · 4 min