AI Papers of the Week
Every paper worth reading in AI, hand-picked one week at a time.
RAG for LLMs
A broad survey of Retrieval-Augmented Generation research, organizing the rapidly growing literature into a coherent map.
BabyLLM Challenge Findings
Reports results from a challenge on sample-efficient pretraining using a developmentally plausible corpus.
QuIP#
Cornell's QuIP# is a 2-bit LLM quantization scheme that combines lattice codebooks with incoherence processing to close the quality gap to FP16.
Gemini 1.0
Google launches Gemini 1.0, a multimodal family natively designed to reason across text, images, video, audio, and code from the ground up.
Advancing Long-Context LLMs
A survey of methodologies for improving Transformer long-context capability across pretraining, fine-tuning, and inference stages.
Survey on Language Models for Code
A comprehensive survey of LLMs for code covering 50+ models, 30+ evaluation tasks, and 500 related works.
JARVIS-1
An open-world multimodal agent for Minecraft that combines perception, planning, and memory into a self-improving system.
S-LoRA
S-LoRA enables serving thousands of LoRA adapters concurrently on a single GPU through memory-paging and custom CUDA kernels.
YaRN (Efficient Context Extension)
YaRN is a compute-efficient method for extending the context window of LLMs well beyond their pretrained length.
FP8-LM
Microsoft's FP8-LM demonstrates that most LLM training variables - gradients, optimizer states - can use FP8 without sacrificing accuracy.
LLMs Meet New Knowledge
A benchmark that evaluates how well LLMs handle new knowledge beyond their training cutoff.
LLMs for Software Engineering
A comprehensive survey of LLMs for software engineering covering models, tasks, evaluation, and open challenges.
Ring Attention
UC Berkeley's Ring Attention scales transformer context to 100M+ tokens by distributing blockwise self-attention across devices in a ring topology.
MemWalker
MemWalker treats the LLM as an interactive agent that traverses a tree-structured summary of long text.
Retrieval Meets Long-Context LLMs
NVIDIA's study comparing RAG and long-context LLMs, with the punchline that the two are complementary rather than substitutes.
Effective Long-Context Scaling (Meta)
Meta proposes a 70B long-context LLM that surpasses GPT-3.5-turbo-16k on long-context benchmarks.
LongLoRA
An efficient LoRA-based fine-tuning recipe for extending LLM context windows without expensive full fine-tuning.
The Rise and Potential of LLM-Based Agents
A comprehensive survey of LLM-based agents covering construction, capability, and societal implications.
Agents Library
An open-source library for building autonomous language agents with first-class support for planning, memory, tools, and multi-agent communication.
Cognitive Architectures for Language Agents (CoALA)
Princeton proposes CoALA, a systematic framework for understanding and building language agents.
Code Llama
Meta releases Code Llama, a family of code-specialized LLMs built on top of Llama 2.
Giraffe
A family of context-extended Llama and Llama 2 models, along with an empirical study of context-extension techniques.
LLM-Based Autonomous Agents Survey
A comprehensive survey of LLM-based autonomous agents covering construction and applications.
L-Eval
A standardized evaluation suite for long-context language models.