Two New Tactile Sensing Papers Tackle the Same Problem From Opposite Directions

The second paper, TacSE3, takes a completely different approach. Rather than treating the tactile image as a geometric registration problem, it converts the image into what the authors call a "decoupled three-dimensional force field." Translation comes from tracking the contact centroid. Rotation, and here's the key difference, comes from analyzing shear patterns in the gel deformation. The physics of friction and shear contain rotational information that pure geometry misses.

TacSE3 also makes a practical concession that InvariantCloud doesn't: it uses paired sensors, one on each fingertip. The paper explicitly notes that dual-sensor sensing "reduces translation-rotation ambiguity." That's an honest admission. A single sensor looking at a low-texture surface genuinely cannot disambiguate certain motions. Two sensors with different viewpoints can.

So which approach is better? It's too early to say definitively, and anyone claiming otherwise is selling something. But I can identify the tradeoffs.

InvariantCloud's geometric approach has a major advantage: it's sensor-agnostic in principle. Any vision-based tactile sensor with embedded markers could use this method. The marker constellation becomes a kind of fingerprint. But it requires those markers to be visible and trackable, which means the gel needs to deform in ways that don't obscure them. Heavy loads, certain object textures, or gel degradation over time could all interfere. The paper doesn't discuss failure modes in detail, which makes me nervous.

TacSE3's physics-based approach is more interpretable. You can trace the rotation estimate back to specific shear patterns, which matters for debugging and for downstream control policies. The paper mentions that their compensation signal improves "disturbance tolerance in downstream manipulation tasks without retraining the base policy." That's a meaningful claim for practical deployment. But the dual-sensor requirement doubles your hardware complexity and calibration burden. And the shear-to-rotation mapping may not generalize across object materials or grip forces.

Look, there's a pattern in robotics research that I've watched play out repeatedly: geometric methods and physics-based methods leapfrog each other for a few years, then someone figures out how to combine them and that hybrid approach wins. I'd bet money that the eventual production-ready solution for tactile pose tracking will use marker constellations for coarse re-localization and shear analysis for fine-grained tracking between keyframes. Neither paper goes there, but the pieces are now on the table.

What neither paper addresses adequately is the manufacturing question. Vision-based tactile sensors are still largely lab-built devices with significant unit-to-unit variation. InvariantCloud's globally invariant registration assumes your marker constellation is consistent across sensors. Is it? TacSE3's force field decomposition assumes consistent gel properties and camera calibration. Real production sensors from companies like GelSight or DIGIT have tolerances that these papers don't account for.

There's also the speed question. Neither paper provides detailed latency numbers that I could find. For real-time manipulation, you need pose estimates at 100Hz or faster. Point cloud registration and force field decomposition are both computationally expensive. Can these run on edge hardware in a gripper, or do they require a tethered workstation? The papers are silent on this, which suggests the answer might be unflattering.

I should note what we don't know yet: how these methods perform on objects that aren't rigid. Both papers focus on rigid body motion estimation. But real manipulation tasks involve deformable objects, soft materials, objects that compress or shift under grip force. The fundamental assumptions of both papers (invariant constellations, interpretable shear patterns) may not hold for a ripe tomato or a foam ball.

The broader context here is that tactile sensing is having a moment. The push toward imitation learning and vision-language models for robot control has exposed a gap: these systems are great at high-level planning but terrible at contact-rich manipulation. You can't learn to thread a needle from video demonstrations alone. The proprioceptive feedback from touch is necessary. Both InvariantCloud and TacSE3 are responses to that need, trying to extract richer information from sensors that already exist rather than inventing new hardware.

That's the right instinct. The vision-based tactile sensor form factor is good enough. GelSight-style sensors are manufacturable, reasonably robust, and provide dense contact information. The bottleneck is software: turning those images into useful state estimates. These two papers represent meaningful progress on that bottleneck, even if neither is the final answer.

My prediction: within 18 months, someone will publish a hybrid approach that uses InvariantCloud-style registration for initialization and loop closure, with TacSE3-style shear tracking for inter-frame estimation. That paper will cite both of these. And it still won't solve the manufacturing variation problem, which will remain the actual blocker for production deployment. The research community loves algorithmic elegance. Industry needs robustness to sensor variation. Those are different problems.

For now, both papers are worth reading if you're building manipulation systems. InvariantCloud if you're working with high-quality sensors and can guarantee marker visibility. TacSE3 if you need interpretable outputs and can afford dual sensors. Neither if you're trying to ship a product next quarter. The real test is always production volume, and we're not there yet.

Mark Kowalski · 10 hours ago · 5 min