Benchmarking Agents on Hard CLI Tasks

Terminal-Bench 2.0 presents a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows. Each task features a unique environment, human-written solution, and comprehensive tests for verification, addressing the gap where current benchmarks either don't measure real-world tasks or aren't sufficiently difficult. - **Challenging for frontier models:** Frontier models and agents score less than 65% on the benchmark, demonstrating that Terminal-Bench meaningfully measures capabilities that current systems struggle with, unlike saturated benchmarks. - **Real-world task inspiration:** Tasks are derived from actual command-line workflows, ensuring the benchmark measures practical skills rather than artificial puzzles, with each task featuring unique environments reflecting diverse real scenarios. - **Comprehensive verification:** Every task includes human-written solutions and comprehensive tests for verification, enabling reliable and reproducible evaluation of agent performance on terminal-based tasks. - **Error analysis insights:** The authors conduct detailed error analysis to identify specific areas for model and agent improvement, providing actionable guidance for researchers developing more capable CLI agents. - **Open evaluation infrastructure:** The dataset and evaluation harness are publicly available at tbench.ai, enabling developers and researchers to benchmark their systems and contribute to advancing agent capabilities in terminal environments.