AutoData: Agentic Search for Pre-training Data Selection
Yan Meng and colleagues frame pretraining data selection as heuristic engineering over per-document features and introduce AutoData, an agent that searches directly over executable selection algorithms.
Ask this paper
The search space is programs, not mixture weights. Scoring, stratification and stochastic selection rules over lexical statistics, categorical labels and perplexity, which lets the agent discover feature interactions instead of reweighting fixed domains.
Validation feedback from a proxy model drives refinement. The agent iteratively edits algorithms against that signal, which is what makes an overnight budget sufficient.
An overnight search beats human-designed pipelines. The discovered selection algorithm outperforms existing curation pipelines on the proxy setting.
The recipe transfers upward. Despite being searched only on a small proxy, it improves the downstream CORE metric at larger scale, which is the claim that makes the approach worth the compute.
Abstract
LLM agents have recently shown promise in automating machine learning engineering by editing model and training code under execution feedback. Data, however, remains largely outside this agentic optimisation loop. We frame pre-training data selection as heuristic engineering over per-document features, i.e., lexical statistics, categorical labels, and perplexity. We introduce AutoData, an agent that searches directly over executable selection algorithms. Unlike prior data mixture methods that optimise weights over a fixed set of domains, AutoData searches a richer program space of scoring, stratification, and stochastic selection rules, discovering feature interactions automatically by iteratively refining algorithms with validation feedback from a proxy model. Within an overnight search, AutoData discovers a selection algorithm that outperforms existing human-designed curation pipelines. Despite being searched only on this small proxy, the discovered recipe transfers to larger scales and improves the downstream metric CORE. These results suggest that data engineering can be treated as an agentic machine learning problem, extending autonomous research from model and training-code optimization to the data.