3D Geometry Is Having a Moment in Robot Manipulation Research. Here's What's Actually New.
A cluster of recent papers is converging on the same insight: point clouds and Fourier-encoded geometry unlock precision that RGB-only policies simply cannot match.
By
·11 hours ago·読了 11 分
Picture a robot arm hovering over a workbench, trying to slot a connector into a circuit board. The connector is roughly 8mm wide. The robot's camera sees it clearly enough. But "seeing" and "knowing where things are in three dimensions" are not the same problem, and the gap between them has been quietly sabotaging robotic manipulation for years.
That gap is what a cluster of recent preprints is trying to close, each from a slightly different angle. Taken together, they sketch a coherent picture of where the field is moving: away from RGB-only policies, toward richer geometric representations, and toward architectures that can actually exploit that geometry at inference time. It is worth unpacking what each contribution actually offers, and where the hype outpaces the evidence.
The core problem is well understood. Standard RGB-based imitation learning policies suffer from depth ambiguity: a pixel tells you colour and intensity, but not how far away the corresponding surface is. Perspective distortion compounds this. A 10mm displacement near the camera looks very different from the same displacement at arm's length, and a naive convolutional or transformer-based policy has to learn to account for that from data alone.
Point clouds sidestep much of this by representing the scene directly in 3D Cartesian coordinates. Each point carries (x, y, z) information, and a policy conditioned on that representation gets a geometric prior essentially for free. The catch, which the literature has known about for some time, is that point cloud-based policies do not reliably outperform image-based ones across all tasks. Performance is, to use the polite phrasing, "highly task-dependent."
関連記事
More in Research
Researchers dropped three path-planning papers in the same week, and together they sketch out something that's been missing from robotics for a long time.
Mark Kowalski · 4 hours ago · 6 min
Sim-to-real gaps, sidewalk autopilots, and egocentric motion maps all landed on arXiv this week. Here is what each actually contributes, and what remains unresolved.
Aisha Patel · 6 hours ago · 9 min
Two new research papers suggest autonomous agents can build and pass their own embodied AI benchmarks. That should make you nervous, not excited.
Mark Kowalski · 9 hours ago · 7 min
PLUME and WEAVER tackle different problems in robotic manipulation, and both papers have results that hold up under scrutiny. Here's what's actually new.
Why? A new paper from a team at arXiv offers a specific and testable hypothesis.
The paper, "Fourier Features Let Agents Learn High Precision Policies with Imitation Learning" (arXiv:2606.12334), argues that the inconsistency of point cloud policies traces back to the spectral bias of neural networks. Neural networks, as has been documented in the theoretical ML literature for several years now, preferentially learn low-frequency functions. When your input features are raw Cartesian coordinates, which change slowly and smoothly across a scene, you are feeding a low-frequency signal into an architecture already predisposed to ignore fine-grained variation. The result is a policy that is geometrically aware in a coarse sense but blind to the millimeter-scale detail that precision manipulation requires.
The proposed fix is conceptually simple: map the Cartesian point cloud into a high-dimensional Fourier feature space before feeding it to the encoder. This is not a new idea in the broader ML literature. Random Fourier features for kernel approximation go back to Rahimi and Recht (2007), and positional encodings in neural radiance fields use essentially the same principle. What is genuinely new here is the systematic validation of this approach specifically for imitation learning policies conditioned on point clouds, across multiple encoder architectures and two substantial benchmarks: RoboCasa and ManiSkill3.
The results are, actually, the research shows this fairly clearly, quite consistent. Fourier features improve performance across diverse encoder architectures, and the gains are robust to hyperparameter variation, which matters a great deal for practical adoption. The authors also validate on a real robot setup, not just simulation, which is the minimum bar I would want to see for any manipulation claim.
I will note one methodological point: the paper does not report the number of real-robot trials or the specific tasks used in physical experiments. Simulation benchmarks are reproducible and well-controlled; real-robot results with small trial counts are not, and it remains unclear from the abstract how much weight to place on the physical validation versus the simulation numbers.
Still, this is genuinely useful work. The core insight, that spectral bias is a concrete and fixable bottleneck for point cloud policies, is the kind of precise mechanistic explanation the field benefits from.
A second paper takes a related approach and applies it to a harder problem. "GeoHAT: Geometry-Adaptive Hybrid Action Transformer for Mobile Manipulation" (arXiv, arXiv:2606.13394) addresses whole-body mobile manipulation, where the robot must coordinate a moving base and a manipulator arm simultaneously, under continuously shifting viewpoints.
This is a meaningfully harder setting than tabletop manipulation, and the paper's design reflects that complexity. GeoHAT uses a lightweight Fourier spatial encoder (the connection to the previous paper is not coincidental; this is a convergent solution to the same spectral bias problem) to map per-pixel 3D coordinates into geometric tokens. Those tokens are then injected into features from a pretrained vision foundation model through a gated fusion mechanism that is modulated by depth validity. The gating is important: noisy or unreliable depth estimates get down-weighted, which prevents the geometric signal from corrupting the semantic prior the foundation model brings.
For action generation, the paper introduces what it calls a Hybrid Whole-Body Action Decoder, which treats arm and base as distinct subspaces with separate cross-attention to their respective relevant visual context. The motivation is sensible. The base needs to know about large-scale spatial layout; the arm needs to know about local geometry near the end effector. Conflating them into a single action vector, as many prior methods do, forces the policy to implicitly learn that separation from data.
On the ManiSkill-HAB simulation benchmark, GeoHAT achieves a 79.3% mean success rate, which the paper reports as a 23.7 percentage point improvement over the strongest baseline. That is a large margin. I would want to see this replicated by an independent group before treating it as definitive, but the architectural reasoning is sound and the ablation structure (I am inferring this from the abstract; the full paper would need to be read carefully) appears to isolate the contributions.
Real-world experiments are also reported, with consistent improvements over baselines. Again, the sample sizes and specific tasks are not detailed in the abstract. This is a recurring limitation of evaluating robotics work from abstracts alone.
A third paper takes a different approach entirely. "AssemLM: A Spatial Reasoning Multimodal Large Language Model for Robotic Assembly" (arXiv, arXiv:2604.08983v2) is not primarily a manipulation policy paper. It is an attempt to build a multimodal language model that can reason about 3D geometry well enough to predict 6D assembly poses, integrating assembly manuals, point clouds, and textual instructions.
This is worth flagging as a distinct contribution. Most vision-language models for robotics operate on 2D perception and struggle with precise 3D spatial reasoning. The authors argue, correctly in my view, that coarse 2D grounding is simply insufficient for assembly tasks where you need to know not just where an object is but how it is oriented in all six degrees of freedom.
AssemLM addresses this by using a specialised point cloud encoder that extracts geometric and rotational features, which are then integrated with the language model's reasoning pathway. The paper also introduces AssemBench, a benchmark with over 900,000 multimodal samples and 6D pose annotations. The scale of that benchmark is notable. Moving evaluation beyond 2D grounding to full 3D geometric inference is a contribution to the field's infrastructure, not just to the model.
The real-robot evaluations reportedly support the approach for multi-step assembly tasks. Code, models, and the dataset are promised to be made publicly available, which is the right call for a benchmark paper.
I know I am being picky here, but I would want to understand the distribution of assembly tasks in the benchmark, specifically whether they generalise beyond the types of assemblies seen during training. Assembly is a domain where overfitting to specific connector geometries is a real risk.
The fourth paper in this cluster, "GAE: Unleashing Physical Potential of VLM with Generalizable Action Expert" (arXiv, arXiv:2510.03896v2), takes a modular approach that I find architecturally interesting.
The core idea is to separate the reasoning and action generation problems entirely. A vision-language model handles high-level planning and produces sparse 3D waypoints as an interface. A separate module, the Generalizable Action Expert (GAE), takes those waypoints plus real-time point cloud observations and generates continuous action trajectories. GAE is pretrained on 150,000 trajectories from simulation and real-world robots, then frozen. Downstream adaptation requires only fine-tuning the VLM to produce the sparse waypoint interface.
This is incremental over prior modular robot learning work, but the specific contribution of using sparse 3D waypoints as the interface between reasoning and execution is well-motivated. The interface is interpretable, which matters for debugging, and it is compact enough that the VLM does not need to learn low-level motor control. The Action Pre-training, Pointcloud Fine-tuning (APPF) scheme that decouples action dynamics from geometry grounding is a sensible engineering choice.
The claim of strong generalisation across diverse visual domains and camera viewpoints is the one I would scrutinise most carefully. Generalisation claims in robotics are notoriously fragile when tested outside the original evaluation distribution. This is based on the results reported in the paper; independent replication would substantially strengthen the claim.
One paper in this cluster sits somewhat outside the manipulation focus. "QueryOcc: Query-based Self-Supervision for 3D Semantic Occupancy" (arXiv, arXiv:2511.17221v2) is primarily an autonomous driving paper, concerned with learning 3D semantic occupancy from camera images without expensive manual annotation.
It is worth noting because the underlying technical problem, learning rich 3D scene representations from limited supervision, is the same problem that underlies all of the manipulation work above. QueryOcc's approach, using independent 4D spatio-temporal queries and a contractive scene representation that preserves near-field detail while compressing distant regions, is a different solution to the geometry problem than Fourier features, but it is addressing the same fundamental bottleneck. A 26% improvement in semantic RayIoU on the Occ3D-nuScenes self-supervised benchmark is a meaningful number, though the autonomous driving evaluation pipeline is sufficiently different from manipulation benchmarks that direct comparison is not meaningful.
The cross-pollination between autonomous driving and manipulation research on 3D representation is worth watching.
Taken individually, each of these papers is a solid contribution. Taken together, they suggest something more interesting: a convergence on a set of architectural principles for geometric robot learning.
First, raw Cartesian coordinates are a poor input representation for neural policies, and the spectral bias explanation gives us a principled reason to prefer Fourier-encoded or otherwise frequency-enriched representations. Second, depth information should be treated as uncertain and gated accordingly, not fused naively. Third, the interface between high-level reasoning (what to do) and low-level execution (how to move) benefits from being explicitly geometric, specifically sparse 3D waypoints, rather than implicit in a shared latent space.
None of these principles is, strictly speaking, novel in isolation. Fourier features for spatial encoding have been used in NeRF since 2020. Modular robot architectures separating planning from control go back decades. The contribution of this cluster is systematic validation across manipulation-relevant benchmarks and, in several cases, real robot hardware.
This raises questions about, well, multiple things. How much of the performance improvement is the geometric representation itself versus the specific architectural choices each paper makes? The papers use different benchmarks, different baselines, and different evaluation protocols, which makes cross-paper comparison essentially impossible at this stage. A unified evaluation framework would help enormously.
Several things remain unclear from this body of work.
The real-robot validation in all four manipulation papers is limited in scope. Simulation benchmarks like RoboCasa, ManiSkill3, and ManiSkill-HAB are valuable precisely because they are reproducible, but the sim-to-real gap for fine manipulation tasks is not solved. We do not yet know whether Fourier-encoded point cloud policies maintain their advantage over RGB policies when deployed on hardware with realistic sensor noise and calibration error.
The computational cost question is also underexplored. GeoHAT's lightweight Fourier spatial encoder is described as avoiding the overhead of a full 3D vision backbone, but the actual inference latency numbers for whole-body mobile manipulation are not front and centre. For real-time control, this matters.
Finally, the reliance on point clouds assumes reasonable depth sensor quality. Several of these papers acknowledge depth noise as a concern and address it architecturally (GeoHAT's gated fusion, for instance), but it is too early to say how robust these approaches are in genuinely challenging sensing conditions: reflective surfaces, thin objects, transparent materials.
An independent replication of the Fourier features result (arXiv:2606.12334) on additional benchmarks would substantially strengthen the case that this is a general-purpose tool rather than a benchmark-specific trick. The authors have released source code, which makes this feasible.
A head-to-head comparison of the modular approach in GAE against the end-to-end approaches in GeoHAT and the Fourier features paper, on the same tasks, with the same robot, would be genuinely informative. Right now, the field is accumulating results on different benchmarks with different baselines, and it is difficult to know which architectural choices are load-bearing.
And for AssemLM specifically, a detailed analysis of failure modes on out-of-distribution assembly geometries would tell us much more about the limits of the approach than additional success rate numbers on in-distribution tasks.
The geometry problem in robotic manipulation is not solved. But this cluster of papers represents real, methodical progress toward solving it, and the convergence on Fourier-encoded representations in particular is worth paying attention to.