Safety envelopes for embodied agents

Stop trying to make the neural network safe. Make a small, simple, auditable thing around it safe, and let the network propose whatever it likes inside that.

You cannot verify a 3-billion-parameter policy. There is no proof technique that will tell you what it does on an input you have not tried, and there will always be inputs you have not tried. This is not a temporary state-of-the-art problem; it is the nature of the object.

The practical response is to stop trying. Put the safety properties somewhere they can be reasoned about — a small, deterministic, human-readable layer between the policy and the actuators — and let the policy be as inscrutable as it likes inside the envelope that layer defines.

The safety layer must be small enough that a person can read all of it in an afternoon and believe it. If it needs its own machine learning, it is not a safety layer.

Four layers, in order of authority#

1. Hardware limits. Torque ceilings, joint stops, an e-stop that cuts power independently of any software. Not bypassable by anything above it. This is the layer that has to hold when everything else has failed, including the operating system.

2. The reflex layer. A fixed-rate loop, typically running faster than the policy, that enforces kinematic and dynamic constraints on whatever command it receives. No learning, no allocation, no I/O.

3. The envelope monitor. Runs at control rate and evaluates state-dependent predicates: is the end-effector inside the permitted workspace, is the measured force below threshold, is the commanded trajectory going to intersect geometry we said it must not.

4. Runtime assurance. The supervisor that decides which controller is in charge — the learned policy, a conservative fallback, or a stop — based on the monitor's output and on whether the policy looks like it is operating in distribution.

Only layers 2 and 3 need real design work. Layer 1 is procurement and wiring; layer 4 is a small state machine.

The reflex layer#

This is the piece we would write first on any new system, before any policy exists at all.

class ReflexLimiter:
    """Clamp a commanded pose to something the arm can actually do next tick.

    Deliberately dull: no model, no state beyond the last command, no branches
    that depend on the task. Runs at control rate, allocates nothing.
    """

    def __init__(self, cfg):
        self.v_max = cfg.max_linear_velocity        # m/s
        self.a_max = cfg.max_linear_accel           # m/s^2
        self.f_max = cfg.max_contact_force          # N
        self.box = cfg.workspace_box                # (lo, hi) in base frame
        self.prev_cmd = None
        self.prev_vel = np.zeros(3)

    def __call__(self, cmd, state, dt):
        p = np.clip(cmd.position, self.box[0], self.box[1])       # workspace

        if self.prev_cmd is not None:
            dv = (p - self.prev_cmd) / dt
            speed = np.linalg.norm(dv)
            if speed > self.v_max:                                # velocity
                dv *= self.v_max / speed
            da = (dv - self.prev_vel) / dt
            accel = np.linalg.norm(da)
            if accel > self.a_max:                                # acceleration
                da *= self.a_max / accel
                dv = self.prev_vel + da * dt
            p = self.prev_cmd + dv * dt
            self.prev_vel = dv

        if np.linalg.norm(state.wrench[:3]) > self.f_max:         # contact
            p = self.prev_cmd                                     # hold, don't push
            metrics.incr("safety.force_limit")

        self.prev_cmd = p
        return replace(cmd, position=p)

Every clamp increments a counter. Those counters are the highest-signal telemetry in the system: a policy whose commands are being clamped frequently is a policy that is asking for things it should not, and that is worth knowing long before it causes an incident.

Control barrier functions, briefly and practically#

For geometric constraints — stay out of this region, keep this clearance — a control barrier function gives you something better than clipping: a minimally invasive correction.

The idea, stripped of formalism: define a scalar function h(x) that is positive inside the safe set and zero at its boundary. Require that h cannot decrease faster than a rate proportional to its current value. Then, at each step, solve for the command closest to what the policy asked for that satisfies that requirement.

def safe_command(u_desired, x, h, grad_h, alpha=5.0):
    """Smallest deviation from the policy's request that keeps hdot >= -alpha*h.

    In the common case the constraint is inactive and this returns u_desired
    unchanged — that's the property that makes it usable in the loop.
    """
    g = grad_h(x)                       # dh/dx
    slack = g @ u_desired + alpha * h(x)
    if slack >= 0:
        return u_desired                # already safe, no correction
    return u_desired - (slack / (g @ g + 1e-9)) * g

The practical value is that the correction is proportional to how close you are to violating the constraint. Far from the boundary the policy is untouched; approaching it, the command is gently deflected; at the boundary, motion into the unsafe set is impossible. Compared to a hard clamp, the resulting motion is smooth and the policy does not fight the limiter.

alpha sets how aggressively the system is allowed to approach the boundary. Large values are permissive and produce late, sharp corrections; small values keep a wide berth and cost task performance. It is one number, it has a clear physical meaning, and it belongs in the config file.

Knowing when the policy is out of its depth#

Constraint enforcement handles "the command is dangerous". The harder question is "the policy no longer knows what it is doing", which usually precedes the dangerous command by a second or two.

Cheap signals, all of which we would instrument before reaching for anything sophisticated:

None of these is calibrated in any statistical sense. All of them are better than nothing, and combined into a weighted score with a threshold tuned on recorded failures, they catch a useful fraction of incipient failures.

The supervisor#

A small state machine with four states and no cleverness at all:

| State | Entry condition | Behaviour | | --- | --- | --- | | NOMINAL | Monitors clean | Policy commands pass through the reflex layer | | DEGRADED | Uncertainty above threshold | Velocity limits reduced, human notified | | FALLBACK | Monitor violation or stall | Scripted retreat to a known-safe pose | | STOPPED | Force violation, watchdog, e-stop | Controlled stop, hold, require human reset |

Transitions out of NOMINAL are one-directional without an explicit human or timed reset. The most common bug in supervisors of this kind is an automatic recovery path that oscillates: uncertainty crosses the threshold, the system degrades, degrading reduces velocity, reduced velocity lowers uncertainty, the system recovers, and the cycle repeats at 2 Hz. Put hysteresis on every threshold and a minimum dwell time in every state.

Testing the layer, not the policy#

The safety layer is the part of the system that can be tested properly, so test it properly:

• Adversarial command replay. Feed the reflex layer commands designed to violate every limit — instantaneous jumps, sustained maximum velocity, commands outside the workspace — and assert on the output. This is a unit test suite and it should run in CI.
• Fault injection. Kill the inference process mid-chunk. Stall it. Deliver a chunk of NaNs. The correct behaviour for each is defined and testable without hardware.
• Envelope coverage. During evaluation runs, log how close the system came to each limit. Limits that are never approached are probably too loose to be doing anything; limits that are hit constantly are shaping behaviour in ways your policy training does not know about.

That last point deserves emphasis. If the limiter is active 30% of the time, the policy you are evaluating is not the policy you trained — it is the policy composed with the limiter, and the two can diverge substantially. Either loosen the limits or retrain with the limiter in the loop.

What this buys you#

Not a guarantee that nothing goes wrong. A guarantee about the class of things that can go wrong: the robot may do the task badly, slowly, or not at all, but it will not exceed a velocity, a force, or a workspace boundary that you wrote down and a colleague reviewed.

For most deployments, that is the difference between a system that can be put in a room with people and one that cannot.