Make Your LLM Fully Utilize the Context (FILM-7B)
Free while signed in. Answers cite the passages they came from.
FILM-7B targets the lost-in-the-middle problem where long-context LLMs fail to retrieve information buried between the start and end of their input. The authors apply an information-intensive (IN2) training recipe to Mistral-7B that forces uniform attention across the full 32K window.
IN2 synthetic data: Training examples are constructed so the answer requires either fine-grained awareness of a ~128-token segment placed anywhere in a 4K-32K context, or integration of information from two or more such segments.
Position robustness: After IN2 training, FILM-7B retrieves reliably from arbitrary positions across document, code, and structured-data contexts, with forward, backward, and bidirectional retrieval patterns all supported.
Real-world gains with minimal regression: NarrativeQA F1 jumps from 23.5 to 26.9 and short-context capability is effectively preserved (MMLU drops only 59.3 to 59.2), showing the fix is targeted rather than disruptive.
Generalizable recipe: IN2 training is positioned as a cheap, bolt-on remedy that can be applied to existing long-context models to correct position-dependent attention failures systematically.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack