Assessing Language Models on Unsolved Questions
Free while signed in. Answers cite the passages they came from.
The paper introduces a new evaluation paradigm that tests models on real unsolved questions from the wild, rather than on fixed-answer exams. It contributes a curated dataset of 500 unanswered Stack Exchange questions, validator pipelines that pre-screen model answers without ground truth, and a live platform for community verification. The approach targets both difficulty and real-world relevance.
Dataset that is difficult and realistic. 500 questions are mined from ~3M unanswered posts across 80 Stack Exchange sites using a three-stage pipeline: rule-based filters on age, views, and votes, then LLM screening for clarity, difficulty, approachability, and objectivity, followed by expert review. The result skews toward STEM but spans sci-fi, history, linguistics, and more.
Validator design built on the generator–validator gap. The authors show models are better at judging answers than generating them, and leverage this with a hierarchical validator: cycle consistency, fact/logic checks, and correctness checks, repeated with reflection and aggregated by unanimous or pipeline voting.
Open platform for human-in-the-loop evaluation. A public site hosts questions, candidate answers, validator traces, and human reviews, tracks pass rates, and credits resolved questions. It aims for ongoing, asynchronous evaluation where solutions deliver actual value to askers and the broader community.
Early results show the task is hard. With the 3-iter validator, the top model passes only 15% of questions; after human checking of 91 validated items, just 10 were confirmed correct across math, physics, stats, programming, and retrocomputing.
Limitations and scope for domain validators. Oracle-free validation struggles with citations and non-formal domains; stronger, domain-specific verifiers (e.g., proof assistants, execution-based checks) could raise precision but at the cost of generality. The team plans iterative dataset versions, improved validators, and deeper community moderation.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack