Coding Agents with an Obstacle-Aware Harness for Safe Robot Manipulation
Bingxin Xu, Yuzhang Shang, Zhen Dong and Emilio Ferrara evaluate coding agents that write robot controllers under a safety constraint, pairing each manipulation goal with an obstacle the robot must not touch, and find the agent collides in most cases.
Ask this paper
Neither perception nor instruction is the cause. The agent reasons about the obstacle in its traces and the prompt already forbids contact; the constraint simply never becomes a planning priority.
The failure splits into two phases. Along the route the model has no notion of a clearing path and does not replan when a chosen route becomes infeasible; at the contact moment it does not treat contact execution as bound by the same constraint.
SafeHarness adds two obstacle-aware harnesses. Route planning grounds objects as bounding boxes and draws candidate waypoint routes to be verified and replanned before execution; contact execution selects a contact position that itself avoids the obstacle.
71.9 percent task success, 87.5 percent collision avoidance. Above the previous state of the art by 6.5 and 27.0 points respectively, with the larger gain on the safety axis.
Abstract
Coding agents have emerged as a promising paradigm for robot manipulation: a language model writes the robot controller as a program, and agents built in this way now operate robots without robot-specific training.Whether this paradigm is also safe, however, has not been asked. We evaluate coding agent under a safety constraint, where each task pairs a manipulation goal with an obstacle the robot must not touch. The agent pursues the goal but collides with the obstacle in most cases, treating task completion as its sole objective while neglecting safety. The agent reasons about the obstacle in its traces, and the prompt already forbids touching it, so neither perception nor instruction is at fault; the fault lies in the planning, where the stated constraint never becomes a priority. By decomposing manipulation into a route phase and a contact-rich moment, we locate the source of the failure. Along the route, the model cannot prioritize the safety constraint, having no notion of a clearing route and none of replanning once a chosen route becomes infeasible. At the contact, it is unaware that contact execution is bounded by the same constraint. To close this gap, we present SafeHarness, which equips the model with two obstacle-aware harnesses that enable it to prioritize the safety constraint. Obstacle-aware route planning grounds the objects as bounding boxes and draws candidate routes over them as sequences of waypoints. The agent then plans a route in advance, verifies it, replans when necessary, and only then executes it. Obstacle-aware contact execution instead selects the contact position so that the contact itself avoids the obstacle. SafeHarness attains 71.9% task success and 87.5% collision avoidance, surpassing the previous SOTA by 6.5% and 27.0%, respectively. These results are $2.3\times$ and $1.5\times$ those of the same agent without harnesses.