AI Papers of the Week
Every paper worth reading in AI, hand-picked one week at a time.
FlashAttention-2
Tri Dao's follow-up to FlashAttention, dramatically improving attention throughput on modern GPUs.
Retentive Network (RetNet)
Microsoft's proposed foundation architecture aiming to replace Transformer attention for LLMs.
Claude 2
Anthropic's second-generation LLM with a detailed model card on safety, alignment, and capabilities.
LongLLaMA
Extends LLaMA's context length using a contrastive training process that reshapes the (key, value) space.
How Language Models Use Long Contexts (Lost-in-the-Middle)
Shows LLM performance drops when relevant information is in the middle of a long context.
Scaling Transformer to 1 Billion Tokens (LongNet)
Microsoft's Transformer variant scaling sequence length past 1B tokens.
Extending Context Window of LLMs (PI)
Position Interpolation extends LLaMA's context to 32K with minimal fine-tuning (within 1000 steps).
Long-range Language Modeling with Self-Retrieval
Jointly trains a retrieval-augmented LM from scratch for long-range modeling.
LOMO
A memory-efficient optimizer that combines gradient computation and parameter update in one step.
Augmenting LLMs with Long-term Memory (LongMem)
Enables LLMs to memorize long history via memory-augmented adaptation.
Augmenting LLMs with Databases (ChatDB)
Combines an LLM with SQL databases as a symbolic memory framework.
No Positional Encodings (NoPE)
Shows explicit position embeddings aren't essential for decoder-only Transformers.
Fine-Tuning Language Models with Just Forward Passes (MeZO)
A memory-efficient zeroth-order optimizer for LLM fine-tuning.
QLoRA
Tim Dettmers' breakthrough technique enabling 65B LLM fine-tuning on a single 48GB GPU.
Reinventing RNNs for the Transformer Era (RWKV)
Combines parallelizable training of Transformers with efficient RNN inference.
StarCoder
An open-access 15.5B code LLM with 8K context and 80+ programming languages.
Learning to Reason and Memorize with Self-Notes
LLMs that deviate from input to explicitly "think" and memorize.
Scaling Transformer to 1M tokens with RMT
Recurrent Memory Transformer extends BERT's effective context to 2M tokens.
Generative Agents: Interactive Simulacra of Human Behavior
Stanford/Google's landmark paper on LLM-powered social simulations.