Autonomous Research for Open-Ended Problems: A Case Study on Telecom Ticket Retrieval
Junghyun Min (Georgetown University, as a Nokia Bell Labs intern) with Huseyin Uzunalioglu and Mohamed Trabelsi (Nokia Bell Labs) run autonomous research agents on an open-ended industrial problem, telecom ticket retrieval, and compare the outcome with 10 months of human work.
Ask this paper
Open-ended setting: The task leaves freedom in document representation, architecture and training-data generation, unlike the narrow search spaces where autonomous research has mostly been shown.
Result: With minimal supervision the agents reach 90% of the human state of the art (0.34 vs 0.38 Recall@1) in 10 weeks instead of 10 months, at up to $200 per Cursor campaign.
Strengths and gaps: The agents do well at narrow hyperparameter optimization but favor local changes over architectural ones, underuse compute, and require operational overhead.
Open-weight orchestration: GPT-OSS 120B failed to orchestrate sub-agents reliably, succeeding once in dozens of attempts.
Recommendation: Human researchers and autonomous research frameworks should work together.
Abstract
Recent breakthroughs in LLM-based systems and their abilities in problem solving and coding have allowed progress in the AI for Science paradigm, potentially replacing human roles in machine learning (ML) research. However, while several frameworks of fully autonomous end-to-end ML research have been proposed, successful implementations of them are often limited to problems with narrow search spaces, like language modeling or biomedical ML benchmarks. In this paper, we explore how autonomous research can be adapted to solve open-ended, industry-grade ML problems, by considering a case study: telecom ticket retrieval, an open-ended task with degrees of freedom in representation, architecture, and training data generation. We discover that autonomous research for open-ended problems with commercial and open-source agents shows both promise and limitations: while autonomous research can excel in narrow hyperparameter optimization, it lacks human-like intuition and creativity and requires operational overhead. Even with minimal human supervision, autonomous research can reach $90\%$ of state-of-the-art performance (0.34 vs. 0.38 Recall@1) in a much shorter time period (10 weeks vs. 10 months of human work) at a modest cost (up to \$200 per Cursor campaign). Our empirical evidence recommends that human researchers and autonomous research frameworks work together for best results in ML research.