FaulT-Bench: Towards Benchmarking Network Troubleshooting LLM Agents under Unreliable User Tickets
Kuan-Hao Tseng and colleagues (University of Sydney) build FaulT-Bench, 200 network troubleshooting scenarios across eight topologies that include false fault reports and wrong root-cause claims, then show SADE, ReAct, and Claude Code all collapse when the network is actually healthy.
Ask this paper
Benchmarks assume the fault exists: Existing network agent evaluations only feed accurate tickets and always plant a real fault. FaulT-Bench adds genuine faults, false reports, wrong device attribution, and wrong root-cause claims, which is what an on-call queue actually looks like.
Agents cannot say nothing is wrong: All three agents are near-saturated on accurate tickets and robust to misdirection, but degrade sharply on healthy networks, probing until some benign condition gets promoted to a root cause. That is a hallucination failure with an operational cost.
Wording beats content: 72 false-premise tickets are rewritten into five reporter personas with network state held fixed. A confidently wrong report is handled about as well as an accurate one, while a vague underspecified report degrades performance sharply.
Three agents, three failure signatures: One over-diagnoses constantly, another leaves runs unanswered, and costs differ substantially. Aggregate accuracy hides all of it.
Why it matters: The negative case, no fault present, is the one production troubleshooting agents will hit most often and the one nobody benchmarks.
Abstract
LLM-based agents are increasingly proposed for network fault diagnosis, but existing benchmarks evaluate them only on accurate tickets and always assume a fault is present, conditions rarely met in practice. We present FaulT-Bench, a benchmark of 200 troubleshooting scenarios across eight network topologies, five reimplemented from public practitioner labs, spanning genuine faults, false fault reports, incorrect device attribution, and incorrect root-cause claims. To isolate how ticket wording affects diagnosis, we further rewrite 72 false-premise tickets into five reporter personas that vary reporter confidence and verifiable detail one factor at a time, holding the network state fixed. Our automated harness deploys each scenario in Kathará, lets agents interact through the NIKA tool interface, and scores free-text diagnoses with an LLM judge across outcome, fix, and reasoning quality. Evaluating SADE, ReAct, and Claude Code, we find all three are near-saturated on accurate tickets and robust to misdirection, yet degrade sharply when the network is healthy and the ticket is wrong, probing until a benign condition can be promoted to a root cause rather than concluding nothing is wrong. Persona rewrites show that how a ticket is written matters more than what it claims: a confidently wrong report is handled about as well as an accurate one, while a vague, underspecified report degrades performance sharply. The three agents also fail differently, from constant over-diagnosis to unanswered runs, at very different cost. These results position FaulT-Bench as a benchmark for developing agentic systems that can reason reliably over the noisy, unreliable tickets of real-world network troubleshooting.