Two New Benchmarks Show Why Robot Perception Still Has a Long Way to Go

From my time building hardware at Fanuc, I can tell you that fisheye cameras are everywhere in modern robotics. They're cheap, they give you wide field of view, and they're essential for near-field manipulation where you need to see what's directly in front of the gripper. But apparently we've been training depth models on data that doesn't actually represent what these cameras see. That's an ambitious oversight, to put it charitably.

The dataset includes both horizontal and vertical stereo setups with varying baselines and fields of view. They've also released 18,000 LiDAR-derived sparse depth training samples. It's the kind of methodical, infrastructure-building work that doesn't get VC funding but actually moves the field forward.

The darkness problem

The NightSight paper tackles a different but related challenge: how do you get a small aerial robot to navigate in complete darkness without strapping a heavy sensor suite to it?

Their solution is clever. They combine a monocular event camera, a coded aperture lens, and an infrared dot projector. The projected pattern creates depth-dependent blur signatures that encode scene geometry. A convolutional neural network decodes these signatures into dense depth maps.

Here's what caught my attention: they trained the model using only synthetic data generated from a simple planar wall setup, and it generalizes zero-shot to complex real-world scenes. That's... unusual. Most deep learning systems need carefully curated training data that matches deployment conditions. The fact that this works suggests they've found a representation that captures something fundamental about the physics of the imaging system.

The specs:

• L1 error: 7.0cm up to 2.5m range (2.80% error)
• Real-time operation: 20 Hz on an NVIDIA Jetson Orin Nano
• Passive computation (the heavy lifting happens in the optics, not the processor)

For search and rescue drones operating in collapsed buildings or mine shafts, this could be significant. The Jetson Orin Nano weighs about 60 grams and draws maybe 7-15W depending on configuration. That's feasible for a small quadrotor. Whether it's robust enough for real deployment is another question, and the paper doesn't include extensive failure mode analysis. It remains unclear how the system handles reflective surfaces, smoke, or dust, all of which you'd encounter in actual disaster scenarios.

The uncomfortable truth

Both papers point to the same underlying issue: the robotics perception stack is built on benchmarks and training data that don't match reality.

WideDepth exists because nobody had built a proper indoor fisheye depth benchmark. NightSight works because event cameras capture information that traditional frame-based cameras miss, but we haven't had the tools to exploit that properly.

I've been covering industrial automation long enough to be skeptical of claims that any single paper will change everything. These won't. But they're the kind of foundational work that, five years from now, we'll look back on as inflection points. Or we won't, because someone will build something better. That's how research works.

The real test is whether these benchmarks and methods get adopted. WideDepth's project page suggests they're releasing the full dataset, which is good. NightSight's hardware setup (coded aperture lens plus IR projector plus event camera) is reproducible but not trivial. We don't know yet whether other labs will build on this or whether it'll become another interesting paper that gets cited but not used.

What I do know is that the gap between demo videos and deployed systems in robotics is often a perception problem, literally. Robots look great in controlled environments with good lighting and known objects. They fall apart when conditions change. These two papers are chipping away at that gap, one dataset and one clever optical trick at a time.

That's not revolutionary. But it might be more important.

Sarah Williams · 4 hours ago · 4 min