Reinforcing Agents with Collective Skills
Binfeng Xu, Yi Dong, Jan Kautz and colleagues at NVIDIA build Skill2Env, a pipeline that compiles public Agent Skills into executable RL environments for terminal agents, each with programmatic tests and a behavioral rubric drawn from the Skill's own quality criteria.
Ask this paper
Pipeline. A containerized Codex planner reads each SKILL.md bundle, researches related public assets and splits the Skill into workflows; a Codex creator builds each task under sampled axes (archetype, verifier pattern, complexity, persona, tone, expertise), and a host gate requires the oracle solution to pass and a no-op to fail.
Data. About 3.4k license-friendly Skills (SkillHub, MIT or Apache 2.0, near-duplicates removed) yield 7,971 Harbor-format tasks across 13 domains, with software engineering under a quarter of the corpus; generation with GPT-5.6 Sol cost over $90k in API usage.
Outcome-only RL. After 300 DPPO updates with the Pi harness, Qwen3.8-27B rises from 49.4% to 54.1% on Terminal-Bench 2.1 and from 33.4% to 37.7% pass@1 (56.6 to 75.1 mean score) on S2EBench, a hand-verified held-out benchmark.
Rubric reward. Adding a GPT-6 Astra judge term (weight 0.2) gives smaller benchmark gains (50.1% on Terminal-Bench 2.1, 34.7% on S2EBench), which the authors attribute to an untuned setup and to benchmarks that score only outcomes.
Skill alignment. Given the source SKILL.md, a judge prefers the outcome-RL model's trajectories over the base model on 54.5% of 200 tasks (33.5% base) and the rubric-RL model's on 73.0% (24.0% base).
Abstract
We introduce Skill2Env, a human-aligned and scalable data recipe for modern agentic RL. The path to safe superintelligence requires behavioral alignment with collective human values and expertise. Public Agent Skills provide a promising source of this supervision---they represent a task distribution most relevant to human society, linking to real-world artifacts, packaging domain expertise and reusable workflows, and often defining success metrics. We develop a pipeline to turn 3.4k web-crawled and filtered Agent Skills into 8k executable terminal environments with both programmatic tests and be- havioral rubrics. Through extensive RL experiments, we observe promising improvements in general agentic abilities. With merely 300 steps of RL training using a DPPO variant, Qwen-3.8 27B gains 4.7 percentage points on Terminal-Bench 2.1, and 4.3 percentage points on S2EBench, our hand-verified private benchmark featuring real-world agent use cases. Further analysis indicates increased behavioral alignment with original Agent Skills on similar problems.