SkillRL
Free while signed in. Answers cite the passages they came from.
SkillRL introduces a recursive skill-augmented RL framework that bridges the gap between raw experience and policy improvement through automatic skill discovery. Instead of storing noisy raw trajectories, SkillRL distills experience into reusable high-level behavioral patterns and evolves them alongside agent policy during training. - **Hierarchical skill library (SkillBank):** An experience-based distillation mechanism extracts reusable behavioral patterns from raw trajectories and organizes them into a hierarchical skill library. This dramatically reduces the token footprint while preserving the reasoning utility needed for complex multi-step tasks. - **Adaptive skill retrieval:** A dual retrieval strategy combines general heuristics with task-specific skills, selecting the most relevant behavioral patterns based on the current task context. This enables the agent to leverage accumulated knowledge without being overwhelmed by irrelevant experience. - **Recursive co-evolution:** The skill library and agent policy evolve together during RL training. As the agent encounters harder tasks, new skills are extracted and existing ones are refined, creating a virtuous cycle where better skills enable better performance, which generates better training data for skill extraction. - **Strong empirical results:** SkillRL achieves state-of-the-art performance with 89.9% success rate on ALFWorld, 72.7% on WebShop, and an average of 47.1% on search-augmented QA tasks, outperforming strong baselines by over 15.3% while maintaining robustness as task complexity increases.
Hierarchical skill library (SkillBank): An experience-based distillation mechanism extracts reusable behavioral patterns from raw trajectories and organizes them into a hierarchical skill library. This dramatically reduces the token footprint while preserving the reasoning utility needed for complex multi-step tasks.
Adaptive skill retrieval: A dual retrieval strategy combines general heuristics with task-specific skills, selecting the most relevant behavioral patterns based on the current task context. This enables the agent to leverage accumulated knowledge without being overwhelmed by irrelevant experience.
Recursive co-evolution: The skill library and agent policy evolve together during RL training. As the agent encounters harder tasks, new skills are extracted, and existing ones are refined, creating a virtuous cycle where better skills enable better performance, which generates better training data for skill extraction.
Strong empirical results: SkillRL achieves state-of-the-art performance with 89.9% success rate on ALFWorld, 72.7% on WebShop, and an average of 47.1% on search-augmented QA tasks, outperforming strong baselines by over 15.3% while maintaining robustness as task complexity increases.
Abstract
Large Language Model (LLM) agents have shown stunning results in complex tasks, yet they often operate in isolation, failing to learn from past experiences. Existing memory-based methods primarily store raw trajectories, which are often redundant and noise-heavy. This prevents agents from extracting high-level, reusable behavioral patterns that are essential for generalization. In this paper, we propose SkillRL, a framework that bridges the gap between raw experience and policy improvement through automatic skill discovery and recursive evolution. Our approach introduces an experience-based distillation mechanism to build a hierarchical skill library SkillBank, an adaptive retrieval strategy for general and task-specific heuristics, and a recursive evolution mechanism that allows the skill library to co-evolve with the agent's policy during reinforcement learning. These innovations significantly reduce the token footprint while enhancing reasoning utility. Experimental results on ALFWorld, WebShop and seven search-augmented tasks demonstrate that SkillRL achieves state-of-the-art performance, outperforming strong baselines over 15.3% and maintaining robustness as task complexity increases. Code is available at this this https URL.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack