RefactorPlatform: An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring Agents
Aziz Ben Amor and colleagues at Pi School release RefactorPlatform, an evaluation harness that holds the environment fixed and varies one coding-agent design axis at a time on 100 repository-scale RefactorBench tasks.
Ask this paper
Axes held apart. Model backbone (via OpenRouter and GitHub Copilot CLI), execution regime (baseline, retrieval-augmented, multi-agent), and prompt specificity are varied independently inside isolated workspaces with per-task token, diff, and transcript logging.
Chunking choice dominates retrieval choice. AST-aware chunking beats naive token-window chunking by 25 to 30% across prompt modes, while naive retrieval scores below the retrieval-free baseline.
Delegation lost to a lean single agent. A retrieval-augmented single agent passed 86% against 66% for the sub-agent configuration, and no task passed under delegation that failed under retrieval.
Retrieval paid for itself. Accuracy gains absorbed the token overhead, leaving cost per successful refactoring unchanged.
Open source. The harness, telemetry export, and AST-based verification are released for reproduction.
Abstract
Repository-scale refactoring requires coding agents to propagate a single change across many interdependent files without altering program behavior, yet to our knowledge no existing harness isolates the design choices that determine agent success on this task. We present RefactorPlatform, an open-source evaluation harness that holds the environment fixed and varies each design axis explicitly: model backbone (via OpenRouter and GitHub Copilot CLI), execution regime (baseline, retrieval-augmented, and multi-agent), and prompt specificity. Each run executes in an isolated workspace with live terminal streaming, per-task logging of tokens, diffs, and transcripts, AST-based verification, and exportable telemetry for audit and reproduction. Demonstrating the platform on 100 multi-file RefactorBench tasks across four model families, we illustrate the analyses it supports: AST-aware chunking outperforms naive token-window chunking by 25-30% across prompt modes, whereas naive retrieval falls below the retrieval-free baseline; a lean retrieval-augmented single agent (86%) beats the sub-agent configuration we evaluated (66%) on matched tasks with no task passing under delegation that fails under retrieval; and retrieval's accuracy gains absorb its token overhead, leaving cost per successful refactoring unchanged. RefactorPlatform is open-sourced to make refactoring-agent evaluation reproducible and auditable.