Self-Challenging Language Model Agents

Self-generated tasks via dual-agent roles: The agent alternates between a challenger role, where it explores the environment and creates tasks, and an executor role, where it learns to solve these tasks via reinforcement learning. The process is designed to emulate how human annotators interact with tools to design meaningful tasks.

Code-as-Task (CaT) formulation: Each synthetic task includes an instruction, a Python-based verification function, a working solution, and several failure cases. This structure ensures task quality by filtering out trivial, impossible, or non-verifiable tasks using automatic code execution checks.

Strong results in both distillation and self-improvement: SCA improves the Llama-3.1-8B-Instruct model’s success rate from 12.0% to 23.5% when learning from its own tasks. In the distillation setting (using a 70B teacher), SCA lifts performance to 32.2% Pass@1, outperforming the prior PAE baseline across all tool-use environments.

Human annotation and ablation confirm task quality: Tasks generated with CaT significantly reduce false positives and negatives compared to PAE. A detailed analysis shows CaT’s filtering removes flawed tasks while retaining diversity when used with stronger models like Llama-3.1-70B.

Scaling and training dynamics: More diverse tasks (not just more trajectories per task) yield better generalization, emphasizing the importance of broad synthetic coverage. Online RL methods like PPO and GRPO can further boost performance, but at higher tuning and compute cost.