AI Papers of the Week
Every paper worth reading in AI, hand-picked one week at a time.
The Era of 1-bit LLMs (BitNet b1.58)
Microsoft's BitNet b1.58 shows that restricting every weight to the ternary set {-1, 0, 1} can match full-precision FP16 transformers on perplexity and downstream tasks at the same parameter count.
Datasets for LLMs: A Comprehensive Survey
A 180+-page survey that catalogs and analyzes the datasets that underpin modern LLM training and evaluation.
LearnAct
LearnAct lets language agents expand and refine their own action space over time by writing and revising Python functions in response to execution feedback.
EMO: Emote Portrait Alive
Alibaba's EMO synthesizes expressive talking-head videos directly from audio, bypassing the intermediate 3D models or facial landmarks used by prior approaches.
On the Societal Impact of Open Foundation Models
Stanford CRFM's policy paper proposes a rigorous framework for assessing the *marginal* risk of open-weight foundation models relative to closed models and pre-existing technologies.
StarCoder 2
BigCode releases StarCoder 2, an open family of code LLMs at 3B, 7B, and 15B parameters trained on The Stack v2, a much larger and cleaner code corpus than the original StarCoder.
LLMs on Tabular Data: A Survey
A survey that maps how LLMs are being applied to tabular data tasks - a domain historically dominated by gradient-boosted trees and specialized architectures.
PlanGPT
PlanGPT is a domain-specialized LLM framework for urban and spatial planning, built in collaboration with the Chinese Academy of Urban Planning.
Stable Diffusion 3
Stability AI previews Stable Diffusion 3, a suite of image-generation models from 800M to 8B parameters that shifts to a diffusion-transformer backbone with flow matching.
Gemma
Google DeepMind releases Gemma, a family of open models (2B and 7B) built from the same research stack as Gemini and shipped with both base and instruction-tuned variants.
LLMs for Data Annotation
A survey that maps the rapidly growing literature on using LLMs to generate, evaluate, and learn from data annotations.
GRIT
GRIT (Generative Representational Instruction Tuning) trains a single LLM to handle both generative and embedding tasks, switching behavior based on instructions.
LoRA+
LoRA+ is a minimal one-line change to LoRA: use different learning rates for the down-projection (A) and up-projection (B) matrices to restore feature learning at large width.
Back to Basics: Revisiting REINFORCE in RLHF
Cohere researchers argue that PPO is overkill for RLHF and that a simpler REINFORCE-style estimator works better in practice.
Recurrent Memory Finds What LLMs Miss
Introduces BABILong, a new long-context benchmark, and shows that transformers with recurrent memory can handle sequences far beyond vanilla LLMs.
When is Tree Search Useful for LLM Planning?
Ohio State + OSU analyze multi-step LLM planning as a generator/discriminator/planner system and argue that current LLM discriminators make tree search a poor choice in practice.
Chain-of-Thought Reasoning Without Prompting
DeepMind shows that LLMs often *already* emit chain-of-thought reasoning in alternative decoding paths, and that selecting those paths via confidence lifts reasoning accuracy with no prompt engineering.
OpenCodeInterpreter
OpenCodeInterpreter is an open-source family of code-execution LLM systems that iteratively refine code using runtime feedback, closing the gap with GPT-4's proprietary Code Interpreter.
Sora
OpenAI unveils Sora, a text-to-video diffusion-transformer that generates coherent, minute-long 1080p videos from natural-language prompts.
Gemini 1.5
Google DeepMind's Gemini 1.5 is a multimodal MoE LLM that scales context to 1M tokens (10M in research settings) while matching or surpassing Gemini 1.0 Ultra on standard benchmarks.
V-JEPA
Meta's V-JEPA learns visual representations by predicting features in masked video regions, without pretrained image encoders, text, negatives, or reconstruction.
Large World Model (LWM)
UC Berkeley's LWM is an open 7B multimodal model trained on long videos and books that handles context windows up to 1M tokens via RingAttention.
The Boundary of Neural Network Trainability is Fractal
Sohl-Dickstein finds that the boundary between trainable and untrainable hyperparameter configurations looks like a Mandelbrot-style fractal across many architectures.
OS-Copilot
OS-Copilot is a framework for building generalist computer agents that use full OS primitives (browser, terminal, files, multimedia, third-party apps) rather than just web DOMs.