AI Papers of the Week
Every paper worth reading in AI, hand-picked one week at a time.
Spectron
Google's Spectron is a spoken-language model trained end-to-end on raw spectrograms rather than text or discrete audio tokens.
Min-K% Prob (Detecting Pretraining Data)
Proposes Min-K% Prob as an effective detection method for determining whether specific text was in an LLM's pretraining data.
ConvNets Match Vision Transformers
DeepMind shows that strong ConvNet architectures pretrained at scale match ViTs on ImageNet performance at comparable compute.
Branch-Solve-Merge (BSM)
BSM decomposes LLM tasks into parallel sub-tasks via three LLM-programmed modules: branch, solve, and merge.
Llemma
Llemma is an open LLM for mathematics built via continued pretraining of Code Llama on the Proof-Pile-2 dataset.
InstructRetro
NVIDIA introduces Retro 48B, the largest LLM pretrained with retrieval at the time.
FireAct (Language Agent Fine-tuning)
Explores fine-tuning LLMs specifically for language-agent use, demonstrating consistent gains over prompting alone.
Retrieval Meets Long-Context LLMs
NVIDIA's study comparing RAG and long-context LLMs, with the punchline that the two are complementary rather than substitutes.
StreamingLLM
MIT's StreamingLLM enables efficient streaming inference by preserving "attention sinks" - early-sequence tokens that most attention mass flows to.
Training LLMs with Pause Tokens
CMU shows that adding a learnable `<pause>` token during both pretraining and fine-tuning gives the model extra "thinking time" and improves reasoning.
RA-DIT (Retrieval-Augmented Dual Instruction Tuning)
Meta's RA-DIT is a lightweight recipe that retrofits LLMs with retrieval capabilities through dual fine-tuning.
The Reversal Curse
Finds that LLMs trained on "A is B" fail to generalize to "B is A" - a surprisingly deep failure of learning.
Effective Long-Context Scaling (Meta)
Meta proposes a 70B long-context LLM that surpasses GPT-3.5-turbo-16k on long-context benchmarks.
Graph Neural Prompting (GNP)
A plug-and-play method that injects knowledge-graph information into frozen pretrained LLMs.
Vision Transformers Need Registers
Meta researchers identify artifact tokens in ViT feature maps and propose a trivial fix: add dedicated register tokens.
LLaVA-RLHF
Adapts factually augmented RLHF to aligning large multimodal models, reducing hallucination without falling into reward-hacking pitfalls.
Qwen
Alibaba releases the Qwen family of open LLMs with strong tool-use and planning capabilities for language agents.
MentaLLaMA
An open-source LLM family specialized for interpretable mental-health analysis on social media.
Contrastive Decoding for Reasoning
Shows that contrastive decoding, a simple inference-time technique, substantially improves reasoning in large LLMs.
LongLoRA
An efficient LoRA-based fine-tuning recipe for extending LLM context windows without expensive full fine-tuning.
Struc-Bench (LLMs for Structured Data)
Studies how LLMs handle complex structured-data generation and proposes a structure-aware fine-tuning method.
Compositional Foundation Models (HiP)
Proposes foundation models that compose multiple expert foundation models trained on different modalities to solve long-horizon goals.
OWL (LLMs for IT Operations)
Proposes OWL, an LLM specialized for IT operations through self-instruct fine-tuning on IT-specific tasks.
Textbooks Are All You Need II (phi-1.5)
Microsoft's phi-1.5 demonstrates that a 1.3B model trained on "textbook-quality" synthetic data rivals much larger models on reasoning.