Two New Papers Tackle Robot Safety Under Uncertainty, But From Opposite Directions
One uses formal verification to know when exploration is safe; the other asks whether we can skip the safety oracle entirely. Both are wrestling with the same fundamental problem.
画像クレジット: Lottie animation by Centre Robotics (LottieFiles Free, used with credit). · source
The challenge of getting robots to act safely when they don't fully understand their environment has produced two notably different approaches this month, both appearing on arXiv within weeks of each other. The first, from researchers working on quadrotor control, proposes a formal "gatekeeper" framework that only allows exploration when it can be mathematically verified as safe. The second asks a more radical question: what if we could approximate safety feedback without having a safety oracle at all?
To be precise, neither paper claims to have solved the fundamental tension between safety and learning. But they represent two distinct philosophies about how to handle uncertainty, and it's worth understanding what each is actually offering.
Before diving into the papers, some background on what makes this hard. When a robot operates under model uncertainty (meaning it doesn't have perfect knowledge of how its actions will affect the world), it faces a choice. It can plan conservatively, assuming worst-case scenarios to guarantee safety. This works, but leads to overly cautious behavior. A drone that's uncertain about wind conditions might refuse to fly near obstacles at all, even when doing so would be perfectly fine.
The alternative is "dual control," where the robot actively explores to reduce its uncertainty while also pursuing its main objective. The problem is that exploration can be dangerous. Flying closer to an obstacle to better estimate wind effects might mean crashing into that obstacle.
Most existing approaches handle this by adding an exploration bonus to the cost function, essentially bribing the robot to try new things. But this requires tuning a weight parameter, and there's no formal guarantee about when exploration actually helps or whether it compromises safety.
関連記事
More in Research
Two new papers show reinforcement learning works better when we stop pretending AI can figure everything out alone.
Mark Kowalski · 1 hour ago · 6 min
Two new papers show hexapods and transformable drones doing whole-body manipulation, which is the kind of unsexy problem that actually matters.
Robert "Bob" Macintosh · 3 hours ago · 4 min
Millimeter-accurate fisheye depth data and a clever low-light navigation hack both point to the same uncomfortable truth: we've been training robots on the wrong data.
James Chen · 5 hours ago · 4 min
Everyone's chasing humanoid demos, but two new perception papers solve problems that have quietly blocked real-world deployment for years.
The paper titled "A Formal gatekeeper Framework for Safe Dual Control with Active Exploration" (available at arXiv) builds on earlier work on gatekeeper architectures for safety verification. The key contribution, and I want to be careful here because the distinction matters, is not a new planning algorithm but rather a framework for deciding when to allow exploration.
The architecture works as follows. A nominal planner generates candidate trajectories. The gatekeeper then evaluates whether exploration would provide "verifiable improvement" without compromising safety. If it passes this check, the exploratory trajectory is executed. If not, the system falls back to a safe, conservative trajectory.
The verification relies on robust planning methods that consider worst-case realizations of the uncertain parameters. The paper extends previous gatekeeper work to generate trajectories that are both safe and informative, meaning they reduce uncertainty about the model while keeping the overall mission cost within a user-defined budget.
The evaluation uses simulation case studies on quadrotor control with parametric uncertainty. I should note that simulation results, while useful, don't tell us how this would perform on real hardware with unmodeled dynamics. The sample size of test scenarios isn't specified in the abstract, which makes it hard to assess generalization.
The second paper, "Approximating Safety Feedback Without a Safety Oracle via Model Predictive Control" (arXiv), takes a fundamentally different stance. Rather than requiring formal safety guarantees upfront, it asks whether we can use modern simulators to construct a proxy for safety feedback.
The motivation here is practical. Explicit safety constraints are often inaccurate or time-consuming to produce. Hand-labeled safety data has the same problems. But simulators like MuJoCo, Isaac Sim, and others can capture complex interactions that would be difficult to model analytically. The safety constraints are implicit in the physics.
The proposed algorithm approximates safety using two key ideas: reversibility and positive-invariance assumptions on the unsafe state space. Here's how it works. First, a proposed action is projected forward in the simulator to a future state. Then, the Model-Predictive Path Integral algorithm (MPPI) attempts to find a path back to a previous state in the trajectory. If such a path exists, the current state is guaranteed to be outside the unsafe (positive invariant) set.
The intuition is that if you can always get back to where you were, you're probably not in an irreversibly bad situation. It's worth noting that this is an approximation. The paper claims it can "approximate the performance of a safety oracle while avoiding classification of unsafe states as safe," which is the critical property (false negatives are acceptable, false positives are not).
What strikes me about these two papers is how differently they frame the problem. The gatekeeper approach says: we need formal guarantees, and the question is how to get useful exploration within those guarantees. The simulator approach says: formal guarantees require accurate models we often don't have, so let's approximate safety empirically.
Neither is obviously wrong. The gatekeeper framework provides stronger theoretical guarantees but requires that you can actually verify safety formally, which assumes you have a good enough model of the system dynamics. The simulator approach is more flexible but depends on the simulator being a faithful representation of reality.
I know I'm being picky here, but the simulator paper's assumption of positive-invariance on the unsafe state space is doing a lot of work. It essentially assumes that if you enter an unsafe region, you can't escape. This is true for some failure modes (crashing is irreversible) but not others (temporarily exceeding a speed limit isn't permanent). The paper doesn't discuss which types of safety constraints this assumption applies to.
The gatekeeper paper is incremental over previous gatekeeper work, which already handled safety verification. The extension to dual control and active exploration is meaningful, but the core architecture is established. What's new is the specific criterion for when exploration provides "verifiable improvement."
The simulator-based paper is, I think, more novel in its framing. Using reversibility as a proxy for safety isn't entirely new (similar ideas appear in the safe reinforcement learning literature), but applying it specifically to bypass the need for an explicit safety oracle is a useful contribution. The combination with MPPI for path planning is also practically relevant.
Neither paper has been replicated yet, and both rely on simulation results. The gatekeeper paper uses quadrotor dynamics, which are well-understood. The simulator paper doesn't specify its test environment in the abstract. This makes it difficult to assess how well either approach would transfer to more complex, real-world scenarios.
For the gatekeeper approach: How computationally expensive is the verification step? If it takes too long, the robot might miss its window for safe exploration. The paper mentions "user-defined budget" for mission cost, but it's not clear how sensitive the results are to this parameter.
For the simulator approach: How accurate does the simulator need to be? If there's a sim-to-real gap, the reversibility check might pass in simulation but fail in reality. The paper claims to avoid classifying unsafe states as safe, but this guarantee presumably depends on simulator fidelity.
More broadly, I'd want to see both approaches tested on the same benchmark tasks. Right now we have no way to compare their performance directly. The robotics community has been slow to adopt standardized benchmarks for safe control under uncertainty, which makes it hard to evaluate progress in this area.
If I were reviewing follow-up work, here's what would be most useful.
First, real hardware experiments. Simulation is necessary but not sufficient. Quadrotors are a good test platform because the dynamics are reasonably well-understood, but they also have fast time constants that stress-test any safety verification system.
Second, a direct comparison between formal and approximate approaches on the same tasks. When does the extra conservatism of formal methods actually cost you? When does the approximation in simulator-based methods actually fail?
Third, more diverse uncertainty types. Both papers focus on parametric uncertainty (unknown but fixed parameters). What about unstructured uncertainty, like novel obstacles or unexpected interactions? This is where the simulator approach might have an advantage, but it hasn't been demonstrated.
Finally, computational benchmarks. For real-time control, you need to make decisions in milliseconds. Neither paper discusses computational requirements in detail, which makes it hard to assess practical deployability.
These two papers are part of a broader trend in robotics toward taking uncertainty seriously. For years, the field relied on accurate models and precise sensing to sidestep the problem. As robots move into less structured environments, that approach breaks down.
The tension between formal guarantees and practical flexibility isn't going away. The gatekeeper approach offers guarantees but requires models good enough to verify safety. The simulator approach offers flexibility but depends on simulators faithful enough to capture real-world constraints. Neither assumption is always true.
What's encouraging is that both papers are explicit about their assumptions and limitations. This is, actually, the research showing how it should be done. Too much work in this area either overpromises on safety guarantees or handwaves about when approximations will fail. These papers are more honest about the tradeoffs.
The fundamental challenge of safe robot learning under uncertainty remains open. But we're at least getting clearer about what the different approaches can and cannot guarantee, which is progress of a sort.