FollowIR
Free while signed in. Answers cite the passages they came from.
FollowIR is both a benchmark and a training set for teaching retrieval models to follow real-world, instruction-style queries rather than just match keywords.
Benchmark from TREC: Evaluation instances come from TREC shared tasks with hundreds to thousands of labeled documents per query, letting the authors test real instruction adherence.
Professional annotator narratives: The training set repurposes assessor narratives (the plain-language instructions TREC annotators follow) as supervised signal for instruction following.
FollowIR-7B: Fine-tuning a 7B Mistral-based retriever on the new training set yields over 13% improvement on the instruction-following benchmark.
Implication: Retrieval quality is now bottlenecked by instruction understanding rather than basic lexical matching, and FollowIR gives the community a concrete dataset to close that gap.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack