AI Papers of the Week
Every paper worth reading in AI, hand-picked one week at a time.
Phi-3
Microsoft's Phi-3 is a family of small language models (3.8B, 7B, 14B) trained on 3.3-4.8T tokens of heavily filtered web data combined with synthetic data. The flagship phi-3-mini rivals Mixtral 8x7B and GPT-3.5 while being small enough to run locally on a phone.
FineWeb
HuggingFace's FineWeb is a 15 trillion token English web dataset built from 96 CommonCrawl snapshots (2013-2024). In 1.8B-parameter ablations, models trained on FineWeb beat C4, RefinedWeb, Dolma, The Pile, SlimPajama, and RedPajama2 across aggregated benchmarks.
Naturalized Execution Tuning (NExT)
NExT teaches LLMs to reason about program runtime behavior by generating synthetic chain-of-thought rationales over execution traces. The approach bootstraps training data through self-training rather than manual annotation, and the learned reasoning transfers to scenarios where traces are unavailable at inference.
Best Practices and Lessons on Synthetic Data
Google DeepMind's survey-style position paper on synthetic data for LLMs. It covers applications, quality-assurance principles, and the open challenges of factuality, fidelity, bias, and privacy.
Aligning LLMs to Quote from Pre-Training Data (Quote-Tuning)
Quote-Tuning aligns LLMs to quote verbatim from trusted pre-training sources, turning the attribution step from post-hoc fact-checking into a built-in model behavior.
LLM2LLM
LLM2LLM is an iterative data augmentation scheme where a strong teacher LLM generates new training examples targeted at the specific mistakes a student model makes during fine-tuning.
DROID
DROID is an open-source robot manipulation dataset that dramatically expands the diversity of real-world robot demonstrations available for imitation-learning research.
Datasets for LLMs: A Comprehensive Survey
A 180+-page survey that catalogs and analyzes the datasets that underpin modern LLM training and evaluation.
StarCoder 2
BigCode releases StarCoder 2, an open family of code LLMs at 3B, 7B, and 15B parameters trained on The Stack v2, a much larger and cleaner code corpus than the original StarCoder.
LLMs for Data Annotation
A survey that maps the rapidly growing literature on using LLMs to generate, evaluate, and learn from data annotations.
ChemLLM
ChemLLM is a chemistry-specialized LLM with a matched dataset (ChemData) and benchmark (ChemBench) for evaluating chemistry-specific capability.
Survey of LLMs
A survey that maps the landscape of the three dominant LLM families - GPT, Llama, and PaLM - and the shared toolbox used to build and augment them.
Grandmaster-Level Chess Without Search
DeepMind shows that a 270M-parameter transformer trained purely with supervised learning on Stockfish-generated data reaches grandmaster-level chess without any search at inference time.
ALOHA 2
ALOHA 2 is a refreshed low-cost bimanual teleoperation platform from Stanford/DeepMind, designed for large-scale robot-learning data collection.
Self-Play Fine-Tuning (SPIN)
SPIN shows that a supervised fine-tuned LLM can keep improving via self-play alone, without any additional human annotations.
PromptBench
A unified library for comprehensive evaluation and analysis of LLMs that consolidates multiple evaluation concerns under one roof.
Generative AI for Math (OpenWebMath / MathPile)
Releases a diverse, high-quality math-centric corpus of ~9.5B tokens designed for training math-capable foundation models.
BabyLLM Challenge Findings
Reports results from a challenge on sample-efficient pretraining using a developmentally plausible corpus.
Magicoder
Magicoder is a fully open-source code LLM that closes the gap with top commercial code models at only 7B parameters via high-quality synthetic instruction data.
MEDITRON-70B
EPFL's MEDITRON is an open-source family of medical LLMs at 7B and 70B parameters, continually pretrained on curated medical corpora.
Translatotron 3
Google's Translatotron 3 performs speech-to-speech translation using only monolingual data - no parallel corpora required.
In-Context Learning Generalization Limits
Investigates whether transformers' in-context learning can generalize beyond the distribution of their pretraining data.
GPT4All Technical Report
The GPT4All technical report documents the model family and the open ecosystem built around democratizing local LLMs.
Open DAC 2023
Meta releases a large DFT dataset for training ML models that predict sorbent-adsorbate interactions in Direct Air Capture (DAC).