Terminal-RSI: Recursive Self-Improving Terminal Agents via Recursive Self-Rewrite and Policy-Refine
Junyao Yang, Zongxia Li, Yucheng Shi and colleagues (Tencent Hy with university co-authors) train a 27B terminal agent through five alternating stages of verified-trajectory rewriting and PPO refinement, raising its four-suite Terminal-Bench mean from 20.68% to 50.58%.
Ask this paper
Two alternating phases. Recursive Self-Rewrite (RSR) fine-tunes on rewritten verified trajectories; Recursive Policy-Refine (RPR) runs PPO with a critic inherited from the previous stage. The sequence Base, RSR-1, RPR-1, RSR-2, RPR-2 starts from Qwen3.8-27B.
Results. Terminal-RSI-27B scores 86.52% on Terminal-Bench 2.1, 79.00% on Terminal-Bench Hard, 16.95% on 3.0 and 19.84% on 4.0, where Gemini 3.8 Flash scores 19.70% in the paper's reference table. The four-suite mean is 29.90 points above the base model and 16.33 above RSR-1.
Reward accounting. In 4,352 audited trajectories, failed runs received 50.09% of the dense reward mass under the earlier reward, and 48 failures passed every executed check but missed the terminal gate. Success-Prioritized Reward separates strict success from bounded partial credit, and rescoring cuts the failed share to 28.72%.
Rewriting stabilizes RL. RPR-1 initialized from the rewritten-trajectory stage beats direct PPO by up to 29.2 points on Terminal-Bench 2.1 at step 20.
Long-horizon limits. On LHTB strict completion moves from 0/46 to 5/46, and SWR100 success from 3% to 6%, so long tasks remain mostly unsolved.
Abstract
Converting verifier outcomes into useful policy updates and partial progress into complete task resolution is crucial for terminal agents. We present Terminal-RSI, which alternates Recursive Self-Rewrite (RSR) of verified trajectories with critic-inheriting Recursive Policy-Refine (RPR). We formulate Turn-Boundary Skip-Observation GAE (TB-GAE) to retain token-specific baselines while applying decay across interaction turns. Family Dense Reward (FDR) calibrates semantic progress across task families; Success-Prioritized Reward (SPR) separates strict success from bounded partial credit. A Warm-up Critic Model, recursive critic handoff, and stage-specific learning rates coordinate the two refinement phases. The five-stage sequence from Qwen3.8-27B base model through RSR-1, RPR-1, RSR-2, and RPR-2 yields Terminal-RSI-27B: 19.84% on Terminal-Bench 4.0, 86.52% on Terminal-Bench 2.1, 16.95% on Terminal-Bench 3.0, and 79.00% on Terminal-Bench Hard. On LHTB, process reward rises from 0.21 to 0.37 and strict completion from 0/46 to 5/46; SWR100 success doubles from 3.00% to 6.00%. By recursively coupling verified trajectory rewriting with critic-inheriting policy refinement, each retained stage consolidates discovered behaviors and progressively advances the agent from partial local repairs to strict terminal verification.