Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents
Tingzhu Bi, Xun Zhang, Jinglin Li, Ping Wang and Meng Ma (Peking University, Beijing University of Posts and Telecommunications) compare three simple harnesses that prescribe none of the model's calls with five published hand-crafted LLM root cause analysis agents on 90 validated microservice failure cases.
Ask this paper
Comparison. The simple harnesses are one call over an evidence package, a SQL agent and the Codex coding agent. They are compared with five published RCA agents and statistical and causal methods on RCAtlas, across six models and three reasoning efforts.
Results. The simple harnesses reach PR@1 of 65 to 77% on three mid-tier models, against 16 to 53% for the published agents and 6 to 39% for non-LLM methods, on every model where both families ran.
Why prescribed agents lose. The published agents fix between a tenth and all of their model calls in the program, and a case study shows these fixed steps skip the decisive evidence.
Scaling factors. A stronger model mainly improves reliability and a multi-turn harness mainly improves aptitude. More reasoning effort changes accuracy by a few points while multiplying time by up to seven.
Abstract
Root cause analysis (RCA) of microservice failures has attracted many LLM agents whose investigation is hand-crafted, through standard operating procedures, fixed stages and role agents. Each is evaluated on its own data against few baselines, so it is unknown whether this design beats letting a capable model investigate on its own. We compare three simple harnesses that prescribe none of the model's calls (one call over an evidence package, a SQL agent and the Codex coding agent) with five published LLM RCA agents and with statistical and causal methods on RCAtlas, 90 validated cases from eight public datasets, across six models and three reasoning efforts. The simple harnesses are the most accurate (PR@1 65-77% on three mid-tier models), ahead of the published agents (16-53%) and the non-LLM methods (6-39%), on every model on which both families were run. The published agents fix from a tenth to all of their model calls in the program, and a case study shows how these prescriptions skip the decisive evidence. A stronger model mainly improves reliability, a multi-turn harness mainly improves aptitude, and more reasoning effort changes accuracy by a few points at most while multiplying the time by up to seven. Among runs more accurate than the best non-LLM method, every Pareto-optimal run in cost or time uses a simple harness.