The Hidden Problem With Robot Training Data: It's Not Just About Quantity
New research suggests most teleoperated robot demonstrations are technically 'successful' but actually terrible for training AI, and there's finally a way to fix that.
Image credit: Lottie animation by Centre Robotics (LottieFiles Free, used with credit). · source
Most coverage of robot learning focuses on the same thing: scale. More data, more demonstrations, more hours of teleoperation. What gets lost is a more fundamental question that two new papers from arXiv tackle head-on: what makes a robot demonstration actually useful?
The answer, it turns out, is more nuanced than "did the robot complete the task." And if you've ever wondered why some robot learning systems generalize beautifully while others fail on slight variations, the quality of training data is probably the culprit.
Here's the core problem. A novice teleoperator can guide a robot arm to pick up a peg and place it in a hole. Task complete. Success logged. But the trajectory they produced might include three false starts, a near-collision with the table edge, and motion that pushed the robot's elbow joint to 98% of its limit. That demonstration "works" but it's teaching the robot terrible habits.
arXiv published research this week on what the authors call Data Quality Assessment and Feedback, or DQAF. The framework analyzes teleoperated episodes across multiple dimensions: motion smoothness, kinematic limit violations, stalls, and what they term "semantic task progress" (basically, did you take a reasonable path through the subtasks).
The key insight is that binary success/failure feedback is nearly useless for improving operator performance. Telling someone "that worked" doesn't help them understand that their jerky corrections and repeated stalls are poisoning the dataset. The DQAF system instead generates natural language feedback explaining an episode is suboptimal and what specific behaviors to correct.
Related coverage
More in Industrial
The global rush toward generative AI is pulling venture dollars away from emerging markets, and African robotics companies are feeling the pinch.
Aisha Patel · 13 hours ago · 6 min
Two days of demos, talks, and networking won't answer the hard questions about where this industry is actually headed.
Mark Kowalski · 14 hours ago · 4 min
The executives rushing to replace workers with AI agents remind me of the automation fever dreams I watched crash and burn in the '90s.
Robert "Bob" Macintosh · 20 hours ago · 3 min
The French AI darling is chasing manufacturing deals with Airbus and BMW, and I've seen this playbook before.
why
In a pilot study with three novice operators, the one receiving this automated feedback improved faster and produced higher-quality demonstrations sooner than those operating blind. That's a small sample, obviously, and I'd want to see this replicated at scale. But the mechanism makes intuitive sense.
A separate paper tackles an upstream question: forget feedback for a moment, what teaching modality should we be using in the first place?
arXiv compared three approaches in a user study with eight participants across three manipulation tasks:
Kinesthetic guidance: physically moving the robot arm by hand
Joystick teleoperation: using a controller to command end-effector motion
Hand gestures: using tracked hand movements to control the robot remotely
The results were more interesting than I expected. Kinesthetic guidance won on most metrics: shortest demonstration times, lowest NASA-TLX workload scores, and highest replay success on orientation-sensitive and contact-rich tasks. This tracks with what I've seen in industrial settings. There's something about feeling the robot's dynamics directly that helps operators produce cleaner trajectories.
But joystick teleoperation actually performed best on simple peg picking. And hand-gesture teaching, which I'll admit I was skeptical of, performed better than expected. In some cases it achieved results comparable to kinesthetic guidance.
The practical implication: there's no universal best method. Task characteristics matter. Contact-rich assembly? Probably want kinesthetic. Simple pick-and-place at a distance? Joystick might be fine. The researchers note that each modality produces different error patterns, which suggests dataset diversity might actually benefit from mixing approaches.
Let me be precise about the kinesthetic guidance results, because "best" is doing a lot of work in that summary.
Kinesthetic demonstrations were shortest in duration. The workload scores (modified NASA-TLX, a standard measure of perceived effort) were lowest. Replay success, meaning the robot could successfully execute the recorded trajectory, was highest on the more complex tasks.
But eight participants across three tasks is a small study. The authors acknowledge this. I've seen enough spec sheets and research papers to know that small-N studies in controlled environments don't always survive contact with messy real-world deployment. The directional findings are useful; the specific numbers should be held loosely.
The DQAF paper has a similar limitation. Three novice operators in a pilot study is enough to demonstrate the concept works, not enough to quantify the improvement precisely. The validation study comparing the system's judgments against a human reviewer is more robust, but we don't have details on inter-rater reliability or edge cases where the system disagreed with human assessment.
Look, the Physical AI hype cycle is real. Every robotics company is talking about learning from demonstrations, training on human data, building foundation models. What's often missing from those conversations is the data quality problem.
From my time building hardware at Fanuc, I can tell you that the gap between "robot completed task" and "robot completed task in a way that's actually deployable" is enormous. A demonstration that works in a lab might include motions that would cause excessive wear on joints, violate safety margins in a production environment, or simply fail when the target object is positioned slightly differently.
The DQAF framework addresses this by treating data collection as a closed-loop process. Operator records demonstration. System analyzes quality. Operator gets specific, actionable feedback. Next demonstration improves. It's obvious in retrospect, but most current data collection pipelines don't work this way. They accumulate hours of teleoperation and sort out quality issues later, if at all.
The teaching modality research matters for a different reason: it suggests we've been too focused on scaling one approach rather than matching method to task. A dataset of 10,000 kinesthetic demonstrations might be worse than 3,000 kinesthetic plus 3,000 joystick plus 3,000 gesture, depending on the downstream application.
Neither paper addresses how these findings transfer to different robot platforms. The kinematic constraints and dynamics of a Franka arm (commonly used in research) are quite different from a KUKA or ABB industrial robot. Whether the optimal teaching modality changes with platform remains an open question.
The DQAF system's natural language feedback is generated through what the authors describe as converting telemetry into "structured quality assessments." The paper doesn't detail the exact prompting or model used for this conversion, which makes it hard to assess how robust the feedback quality is across different failure modes.
And there's a broader question neither paper tackles: at what scale do these quality interventions actually pay off? If you're collecting 100,000 demonstrations, is it worth the overhead of per-episode feedback? Or does the noise wash out at scale anyway? We don't know yet.
What we do know is that the "just collect more data" approach to robot learning has limits. These papers point toward a more thoughtful alternative: collect better data, match your teaching method to your task, and close the feedback loop so operators actually improve. That's less exciting than announcements about billion-parameter models, but it might matter more for getting robots that actually work.
