AI Papers of the Week
Every paper worth reading in AI, hand-picked one week at a time.
C4AI Command-R
Cohere for AI releases Command-R, a 35B open-weight LLM tuned specifically for retrieval-augmented generation, tool use, and multilingual workflows.
Claude 3
Anthropic releases the Claude 3 family (Haiku, Sonnet, Opus), with Opus leapfrogging GPT-4 on many standard benchmarks and bringing frontier multimodal capability plus a much larger context window.
GaLore
GaLore (Gradient Low-Rank Projection) reduces optimizer-state memory during LLM training while still permitting full-parameter updates, unlike LoRA-style adapters that restrict learning to a low-rank subspace.
Mistral Large
Mistral AI releases Mistral Large, its flagship closed-weight LLM positioned as the second-ranked API-accessible model behind GPT-4 at launch.
The Era of 1-bit LLMs (BitNet b1.58)
Microsoft's BitNet b1.58 shows that restricting every weight to the ternary set {-1, 0, 1} can match full-precision FP16 transformers on perplexity and downstream tasks at the same parameter count.
StarCoder 2
BigCode releases StarCoder 2, an open family of code LLMs at 3B, 7B, and 15B parameters trained on The Stack v2, a much larger and cleaner code corpus than the original StarCoder.
PlanGPT
PlanGPT is a domain-specialized LLM framework for urban and spatial planning, built in collaboration with the Chinese Academy of Urban Planning.
Gemma
Google DeepMind releases Gemma, a family of open models (2B and 7B) built from the same research stack as Gemini and shipped with both base and instruction-tuned variants.
LoRA+
LoRA+ is a minimal one-line change to LoRA: use different learning rates for the down-projection (A) and up-projection (B) matrices to restore feature learning at large width.
Recurrent Memory Finds What LLMs Miss
Introduces BABILong, a new long-context benchmark, and shows that transformers with recurrent memory can handle sequences far beyond vanilla LLMs.
Gemini 1.5
Google DeepMind's Gemini 1.5 is a multimodal MoE LLM that scales context to 1M tokens (10M in research settings) while matching or surpassing Gemini 1.0 Ultra on standard benchmarks.
Large World Model (LWM)
UC Berkeley's LWM is an open 7B multimodal model trained on long videos and books that handles context windows up to 1M tokens via RingAttention.
LLM Agents Can Autonomously Hack Websites
The paper shows GPT-4 agents with tool use and long context can autonomously exploit real websites, including performing blind SQL injection and schema extraction.
DeepSeekMath
DeepSeek releases DeepSeekMath 7B, a math-specialized LLM that closes much of the gap to GPT-4 and Gemini-Ultra on MATH by combining better data and a new RL objective.
LLM-based Multi-Agent Systems Survey
A survey of the fast-growing LLM-based multi-agent systems space, covering both problem-solving applications and "world simulation" research.
Compression Algorithms for LLMs
A survey covering the main families of LLM compression techniques and when each one is appropriate.
Resource-efficient LLMs & Multimodal Foundation Models
A wide-ranging survey of efficiency techniques for LLMs and multimodal foundation models, spanning architecture, algorithms, and system design.
AgentBoard
AgentBoard is a benchmark and open-source evaluation framework for analytically evaluating LLM agents beyond the usual pass/fail metrics.
MoE-Mamba
MoE-Mamba combines state-space models (Mamba) with Mixture-of-Experts to scale LLMs more efficiently than either Mamba or Transformer-MoE alone.
RAISE
RAISE is an advanced agent architecture that adds a dual-memory system on top of a ReAct-style backbone to better support long-running conversational agents.
Fast Inference of Mixture-of-Experts
Achieves practical Mixtral-8x7B inference on consumer hardware through MoE-aware quantization and offloading.
Fact Recalling in LLMs
A mechanistic-interpretability study showing that early MLP layers function as a lookup table for factual recall.
PowerInfer
A high-speed LLM inference engine for consumer GPUs that exploits sparse neuron activation patterns to run large models on commodity hardware.
LLM in a Flash
Apple researchers show how to run LLMs larger than available DRAM by streaming weights from flash storage on demand.