Two New Tactile Sensing Papers Tackle the Same Problem From Opposite Directions
InvariantCloud and TacSE3 both promise better 6-DoF pose tracking for robot grippers, but their approaches reveal a deeper split in how the field thinks about touch.
Crédit photo: Lottie animation by Centre Robotics (LottieFiles Free, used with credit). · source
Most coverage of tactile sensing research treats each new paper as an isolated breakthrough. That misses the more interesting story: two papers released within weeks of each other are attacking the exact same problem, yaw estimation in vision-based tactile sensors, and arriving at fundamentally different solutions. One bets on geometry. The other bets on physics. Both claim superior results. Only one approach will scale.
Let me back up. The problem both teams are solving is deceptively simple to state: when a robot grips an object, how does it know if that object is rotating in its fingers? Vision-based tactile sensors (think camera-equipped silicone fingertips) can detect translation reasonably well by tracking how the contact patch moves. But rotation around the Z-axis, what engineers call yaw, has been a persistent headache. The sensor sees a blob of deformed gel, and that blob looks roughly the same whether the object has rotated 5 degrees or stayed still. Cumulative drift compounds the error over time.
The first paper, InvariantCloud, comes from a team that's clearly frustrated with incremental tracking approaches. Their insight is that the surface markers embedded in tactile sensor gel form constellations that are globally unique. Instead of tracking frame-to-frame motion and accumulating error, they perform one-shot point cloud registration against a reference. It's a bit like how GPS works: you don't dead-reckon your position from your last known location, you triangulate from fixed satellites. The markers are the satellites.
This is clever, and from my time building hardware, I've seen enough tracking systems fail from drift to appreciate the appeal. The paper claims "superior yaw tracking accuracy and re-localization repeatability" compared to existing benchmarks. I'd want to see those benchmarks specified more precisely (superior by how much? under what conditions?) but the core idea is sound. If your marker constellation is truly unique and globally identifiable, you can re-localize at any point without accumulated error.
À lire aussi
More in Research
Two new papers tackle the unsexy engineering problems that'll determine whether robot-assisted surgery actually works at scale.
Mark Kowalski · 4 hours ago · 4 min
Researchers are finding clever workarounds for the hardware that's supposed to be essential. I'm cautiously intrigued.
Sarah Williams · 4 hours ago · 3 min
Two new papers tackle the unsexy but critical problems of actually controlling squishy robots, and it's about time.
Mark Kowalski · 8 hours ago · 5 min
Forget the hype about AI designing robots. These researchers are solving the boring, fundamental problem that's held back the field for decades.
The second paper, TacSE3, takes a completely different approach. Rather than treating the tactile image as a geometric registration problem, it converts the image into what the authors call a "decoupled three-dimensional force field." Translation comes from tracking the contact centroid. Rotation, and here's the key difference, comes from analyzing shear patterns in the gel deformation. The physics of friction and shear contain rotational information that pure geometry misses.
TacSE3 also makes a practical concession that InvariantCloud doesn't: it uses paired sensors, one on each fingertip. The paper explicitly notes that dual-sensor sensing "reduces translation-rotation ambiguity." That's an honest admission. A single sensor looking at a low-texture surface genuinely cannot disambiguate certain motions. Two sensors with different viewpoints can.
So which approach is better? It's too early to say definitively, and anyone claiming otherwise is selling something. But I can identify the tradeoffs.
InvariantCloud's geometric approach has a major advantage: it's sensor-agnostic in principle. Any vision-based tactile sensor with embedded markers could use this method. The marker constellation becomes a kind of fingerprint. But it requires those markers to be visible and trackable, which means the gel needs to deform in ways that don't obscure them. Heavy loads, certain object textures, or gel degradation over time could all interfere. The paper doesn't discuss failure modes in detail, which makes me nervous.
TacSE3's physics-based approach is more interpretable. You can trace the rotation estimate back to specific shear patterns, which matters for debugging and for downstream control policies. The paper mentions that their compensation signal improves "disturbance tolerance in downstream manipulation tasks without retraining the base policy." That's a meaningful claim for practical deployment. But the dual-sensor requirement doubles your hardware complexity and calibration burden. And the shear-to-rotation mapping may not generalize across object materials or grip forces.
Look, there's a pattern in robotics research that I've watched play out repeatedly: geometric methods and physics-based methods leapfrog each other for a few years, then someone figures out how to combine them and that hybrid approach wins. I'd bet money that the eventual production-ready solution for tactile pose tracking will use marker constellations for coarse re-localization and shear analysis for fine-grained tracking between keyframes. Neither paper goes there, but the pieces are now on the table.
What neither paper addresses adequately is the manufacturing question. Vision-based tactile sensors are still largely lab-built devices with significant unit-to-unit variation. InvariantCloud's globally invariant registration assumes your marker constellation is consistent across sensors. Is it? TacSE3's force field decomposition assumes consistent gel properties and camera calibration. Real production sensors from companies like GelSight or DIGIT have tolerances that these papers don't account for.
There's also the speed question. Neither paper provides detailed latency numbers that I could find. For real-time manipulation, you need pose estimates at 100Hz or faster. Point cloud registration and force field decomposition are both computationally expensive. Can these run on edge hardware in a gripper, or do they require a tethered workstation? The papers are silent on this, which suggests the answer might be unflattering.
I should note what we don't know yet: how these methods perform on objects that aren't rigid. Both papers focus on rigid body motion estimation. But real manipulation tasks involve deformable objects, soft materials, objects that compress or shift under grip force. The fundamental assumptions of both papers (invariant constellations, interpretable shear patterns) may not hold for a ripe tomato or a foam ball.
The broader context here is that tactile sensing is having a moment. The push toward imitation learning and vision-language models for robot control has exposed a gap: these systems are great at high-level planning but terrible at contact-rich manipulation. You can't learn to thread a needle from video demonstrations alone. The proprioceptive feedback from touch is necessary. Both InvariantCloud and TacSE3 are responses to that need, trying to extract richer information from sensors that already exist rather than inventing new hardware.
That's the right instinct. The vision-based tactile sensor form factor is good enough. GelSight-style sensors are manufacturable, reasonably robust, and provide dense contact information. The bottleneck is software: turning those images into useful state estimates. These two papers represent meaningful progress on that bottleneck, even if neither is the final answer.
My prediction: within 18 months, someone will publish a hybrid approach that uses InvariantCloud-style registration for initialization and loop closure, with TacSE3-style shear tracking for inter-frame estimation. That paper will cite both of these. And it still won't solve the manufacturing variation problem, which will remain the actual blocker for production deployment. The research community loves algorithmic elegance. Industry needs robustness to sensor variation. Those are different problems.
For now, both papers are worth reading if you're building manipulation systems. InvariantCloud if you're working with high-quality sensors and can guarantee marker visibility. TacSE3 if you need interpretable outputs and can afford dual sensors. Neither if you're trying to ship a product next quarter. The real test is always production volume, and we're not there yet.