Learning the Task or Learning the Verifier? Rethinking Criteria Exposure in Embodied Self-Improvement
Wenbo Huang, Daquan Zhou and colleagues at Peking University and NUS test what a self-improving embodied coding agent learns when it is shown the benchmark's success criteria, a practice they call Criteria Exposure (CE).
Ask this paper
Setup. One self-improvement loop writes robot-control programs with and without access to the benchmark criteria. The authors also write instruction-faithful criteria that check only what the instruction asks.
Real gain. Learning from success alone doubles success from 24.6% to 47.8%, and that gain holds when the criteria change.
Exposure gain. Reading the criteria adds 14.2 more points (62.0%), all of it on tasks where the criteria demand more than the instruction (51.6% vs 27.8%). Under instruction-faithful criteria the lead disappears (45.0% vs 47.2%).
Overfitting. A library that read the faithful criteria scores 65.0% under them and 13.5% under the original ones, so each library mostly learns the verifier it was shown.
Recommendation. Self-improvement claims on embodied benchmarks should be measured without the verifier visible and against criteria that match the instruction.
Abstract
Self-improving embodied agents are increasingly being developed and compared on robot simulation benchmarks. Several recent works expose the benchmark's success criteria to the learning agent, a practice we call Criteria Exposure (CE). These criteria, originally written to verify policies trained by imitation learning, often demand more than the corresponding language instructions, so a score from such a verifier may not faithfully reflect whether the instructed task was completed, especially for robot-control programs written by an agent. As embodied coding agents report ever higher benchmark scores, a natural question arises: when a coding agent is exposed to the benchmark's criteria, does it generate code that better performs the instructed task, or code that simply overfits to the verification criteria? We conduct controlled experiments with a self-improving agent with and without CE, and find that with CE the agent quickly produces code with a higher success rate under the given criteria, but this advantage disappears when the criteria are changed slightly without departing from the original instructions. More importantly, the skills the agent develops under CE transfer poorly across tasks. We further propose instruction-faithful criteria and show that they better reflect the performance of agent-generated programs.