AI Papers of the Week
Every paper worth reading in AI, hand-picked one week at a time.
Llama 3
Meta's Llama 3 launches with 8B and 70B pretrained and instruction-tuned variants. Llama 3 8B beats Gemma 7B and Mistral 7B Instruct, and Llama 3 70B is competitive with Gemini Pro 1.5 and Claude 3 Sonnet on standard benchmarks.
Mixtral 8x22B
Mistral's Mixtral 8x22B is a sparse Mixture-of-Experts model with 141B total / 39B active parameters and a 64K context window, released under Apache 2.0. It leads open models on MMLU and posts strong math, code, and multilingual numbers.
Chinchilla Scaling: A replication attempt
This paper re-examines the third estimation procedure in Hoffmann et al. (2022) Chinchilla scaling law and finds it is inconsistent with the paper's own first two methods, fails to fit the extracted data, and reports implausibly narrow confidence intervals.
How Faithful are RAG Models? (ClashEval)
ClashEval constructs a 1,200-question benchmark across six domains with intentionally corrupted retrieved documents to measure when RAG helps and when it misleads GPT-4 and other top LLMs.
A Survey on Retrieval-Augmented Text Generation for LLMs
This survey organizes the RAG literature into a four-stage framework (pre-retrieval, retrieval, post-retrieval, generation) and traces the paradigm's evolution alongside open challenges.
The Illusion of State in State-Space Models
This paper proves that modern state-space models (Mamba, S4, etc.) share the same expressive ceiling as transformers: they cannot compute anything outside the TC^0 complexity class, despite the RNN-like "state" vocabulary they borrow.
Reducing Hallucination in Structured Outputs via RAG
This paper deploys a compact RAG pipeline - small retriever plus small LM - for an enterprise workflow-generation task and shows it reduces hallucination while improving out-of-domain generalization vs a baseline LLM.
Emerging AI Agent Architectures
A short survey mapping the current landscape of LLM-based agent architectures, focused on reasoning, planning, and tool calling as the three capability pillars for complex agentic workflows.
LM In-Context Recall is Prompt Dependent
Using needle-in-a-haystack tests across multiple models, this paper shows that in-context recall is highly sensitive to prompt wording and that training data biases can silently degrade a model's ability to retrieve from its own context.
A Survey on State Space Models
A comprehensive survey of modern SSMs with a principles-first walkthrough, taxonomy of existing variants, and experimental comparison across NLP, vision, graph, multimodal, point-cloud, event-stream, and time-series tasks.
Leave No Context Behind (Infini-attention)
Google's Infini-attention extends Transformer LLMs to effectively infinite context with bounded memory and compute. It blends a compressive memory module with both masked local attention and linear long-term attention inside a single Transformer block.
OpenEQA
Meta's OpenEQA is an open-vocabulary benchmark for embodied question answering: 1,600+ human-written questions across 180+ real-world environments, with a calibrated LLM-as-judge metric that tracks human agreement closely.
CodeGemma
CodeGemma is a family of open code LLMs built on Gemma, released in 2B (pretrained), 7B (pretrained), and 7B-IT (instruction-tuned) variants. The 2B model is optimized for low-latency code completion, and the 7B-IT model leads its weight class on HumanEval.
LM-Guided Chain-of-Thought
This paper offloads rationale generation to a small, trained LM while keeping a frozen large LM as the answer predictor. The small model is optimized with knowledge distillation and reinforcement learning so it produces rationales that steer the large model more effectively.
Best Practices and Lessons on Synthetic Data
Google DeepMind's survey-style position paper on synthetic data for LLMs. It covers applications, quality-assurance principles, and the open challenges of factuality, fidelity, bias, and privacy.
Reasoning with Intermediate Revision and Search (THOUGHTSCULPT)
THOUGHTSCULPT is a graph-based reasoning framework that combines Monte Carlo Tree Search with an explicit revision action, letting an LLM iteratively rewrite earlier thoughts instead of only extending them.
Overview of Multilingual LLMs
A first-of-its-kind survey on multilingual LLMs, organized by multilingual alignment principles rather than model-family hierarchy. The authors propose a unified taxonomy and collect open resources to accelerate future research.
The Physics of Language Models
This paper measures how many bits of factual knowledge a language model can store per parameter and finds a remarkably stable 2-bits-per-parameter ceiling, even after int8 quantization. A 7B model can therefore hold ~14B bits - more than the English Wikipedia and textbooks combined.
Aligning LLMs to Quote from Pre-Training Data (Quote-Tuning)
Quote-Tuning aligns LLMs to quote verbatim from trusted pre-training sources, turning the attribution step from post-hoc fact-checking into a built-in model behavior.
The Influence Between NLP and Other Fields
This EMNLP 2023 analysis quantifies NLP's cross-disciplinary engagement using a Citation Field Diversity Index across 23 academic fields. The headline: NLP has become dramatically more insular over four decades.
Many-shot Jailbreaking
Anthropic shows that long-context windows enable a new attack where hundreds of fake user/assistant dialogues are packed into a single prompt, coaxing frontier LLMs to answer the final harmful question despite safety training.
SWE-Agent
Princeton's SWE-agent pairs a language model with a custom agent-computer interface (ACI) that exposes file navigation, editing, and test execution as discrete tools, letting the agent autonomously resolve real GitHub issues.
Mixture-of-Depths
DeepMind proposes dynamically allocating transformer FLOPs across sequence positions via a top-k router, so "easy" tokens skip expensive blocks while "hard" tokens get full computation.
Long-context LLMs Struggle with Long In-Context Learning
LongICLBench stress-tests 13 long-context LLMs on extreme-label classification with up to 174 classes and 50K-token prompts, exposing sharp quality cliffs beyond 20K tokens.