Learning Generalizable Behaviors for Terminal Agents
Yihang Yao, Bo Pang, Semih Yavuz and colleagues at Salesforce AI Research, with Ding Zhao at Carnegie Mellon, study what RL actually changes in terminal agents and propose RIVER, a training recipe that improves reward quality by filtering defective environments and penalizing repetitive loops.
Ask this paper
Behaviors over skills. Under their agentic compositional generalization hypothesis, RL mainly shapes multi-turn behaviors such as inspecting before acting, verifying before declaring completion and abandoning a failing approach, which compose skills the model already has from pre-training and SFT. Verify-before-done and repetition rate are the two behavior features that best predict task success.
Audit of public environments. In the cleanest public pool audited (TMax) only 35.8% of environments are clean and 40.4% have verifiers that are too weak; TermiGen is 10.1% clean and TerminalTraj-5k 3.3%. Defects corrupt rewards in both directions, giving reward 1 for copying leaked answers and reward 0 to correct solutions.
RIVER recipe. Environments are screened with rubrics and a pass@2 oracle check, leaving RIVER-TMax-3.5K, and a turn-level penalty is applied when a turn's command and observation both have Jaccard similarity above 0.8 with an earlier turn.
Results. River-8B averages 19.4 across Terminal-Bench-Lite, Terminal-Bench v2.1, Terminal-Bench-Pro and Terminal-World-Verified, the best of the open RL-trained 8B models evaluated on all four, against 17.8 for OpenThinker-8B-RL and 17.7 for RL on 3.5K randomly sampled TMax environments.
Scaling. Using fewer than 30% of TMax's environments, RIVER increases RL gains by 106% on Terminal-Bench-Lite and 30% on Terminal-Bench v2.1 on average for models from 2B to 27B, and the recipe carries across model families, harnesses and RL objectives.
Abstract
Terminal agents are a compelling application of large language models (LLMs), with the potential to integrate deeply into users' daily workflows. Reinforcement learning (RL) is a key technique for improving their capabilities, making scalable training environments a central challenge. Since public real-user interaction data are scarce, synthetic environments provide a practical alternative, but often suffer from domain gaps and limited fidelity, leading to poor generalization. Existing work mainly scales the quantity and diversity of synthetic environments, while reward-signal quality and the mechanisms governing generalization remain under-explored. We study how RL improves terminal agents and propose the Agentic Compositional Generalization hypothesis: rather than teaching new domain-specific skills from scratch, RL primarily shapes high-level decision-making behaviors that compose and route low-level skills acquired during pre-training and supervised fine-tuning (SFT). This account is consistent with our empirical results and suggests that verifier quality, which determines which behaviors are reinforced, is more important than simply increasing environment quantity or diversity. Motivated by this insight, we propose River, a simple training recipe that improves reward quality by filtering low-quality environments and augmenting outcome rewards with process-level behavior regularization. Using this recipe, our RL-trained agent achieves the best performance among evaluated open-source RL-trained 8B models across four terminal-agent benchmarks. River also generalizes across model families, scales, agent harnesses, and RL objectives. Using fewer than 30% of the TMax training environments, River improves RL gains by 106% and 30% on average for models ranging from 2B to 27B on Terminal-Bench-Lite and Terminal-Bench-v2.1, respectively.