Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive Optimization
Sihan Ge and colleagues at Cardinal Operations and Shanghai Jiao Tong University benchmark whether an LLM agent knows when to ask a clarifying question before turning a natural-language operations research request into a mathematical program.
Ask this paper
The gap in existing evaluation: Prior work assumes a complete specification, so it never tests whether the agent recognizes a missing objective, constraint or business rule that would change the resulting program.
OR-Clarify: Each task gives a partial public problem description, withholds structured hidden slots, and evaluates agents through bounded interaction with a simulated user. It supports open-ended and choice-based clarification and measures slot recovery, stopping behavior, silent assumptions, and interaction cost.
InterOPT: A two-stage framework that first identifies unresolved formulation-critical gaps, then uses them to decide whether to ask another question or stop.
Results: In the choice-based setting InterOPT substantially outperforms all baselines on exact slot recovery; in the open-ended setting it stays competitive with strong prior methods. Measuring silent assumptions separately from slot recovery is what distinguishes an agent that stopped because it was done from one that stopped and guessed.
Abstract
Large language models (LLMs) are increasingly used to formulate optimization models from natural-language problem descriptions, yet realistic operations research (OR) requests are often incomplete: missing objectives, constraints, or business rules can change the resulting mathematical program. Existing evaluations largely assume a complete specification and therefore overlook whether an agent knows when clarification is needed before modeling. We introduce OR-Clarify, a benchmark for pre-formulation clarification. Each task presents a partial public problem description, withholds structured hidden slots, and evaluates agents through bounded interaction with a simulated user. The benchmark supports both openended and choice-based clarification, and measures slot recovery, stopping behavior, silent assumptions, and interaction cost. We further propose Interactive Optimization (InterOPT), a two-stage framework that identifies unresolved formulation-critical gaps and uses them to guide whether to ask the next question or to stop. In our choice-based experiments, InterOPT substantially outperforms all baselines in exact slot recovery; in the open-ended setting, it remains competitive with strong prior methods. Together, OR-Clarify and InterOPT reframe OR assistance as a selective completeness decision: clarify when needed, stop when ready, and quantify what remains missing.