AI Papers of the Week
Every paper worth reading in AI, hand-picked one week at a time.
Visualization-of-Thought
Microsoft's Visualization-of-Thought (VoT) prompts LLMs to emit intermediate "mental images" of their reasoning state, lifting spatial-reasoning accuracy on grid-world tasks and beating multimodal baselines that actually see images.
The Unreasonable Ineffectiveness of the Deeper Layers
The paper shows that open-weight LLMs tolerate removing up to half of their transformer blocks with only minor degradation, provided a short QLoRA pass is used to heal the damage afterwards.
JetMoE
MyShell's JetMoE-8B is an open MoE model trained for under $100K that matches or beats LLaMA2-7B, showing that competitive LLM training can be achieved on modest budgets with public data.
ReFT: Representation Finetuning for LMs
Stanford's ReFT freezes the base model and instead learns small interventions on hidden representations at selected layers, offering a more parameter-efficient alternative to LoRA-style PEFT.
Advancing LLM Reasoning (Eurus)
OpenBMB's Eurus is a suite of reasoning-specialized LLMs (7B and 70B) fine-tuned on UltraInteract, a new alignment dataset built around preference trees for complex math, code, and logical tasks.
Training LLMs over Neurally Compressed Text
The paper proposes Equal-Info Windows, a neural compression scheme that segments text into equal-bit-length blocks so an LLM can train directly on compressed bytes without losing learnability.
DBRX
Databricks releases DBRX, a 132B-total / 36B-active open Mixture-of-Experts LLM that beats established open models on MMLU, HumanEval, and GSM8K while delivering 2x faster inference than LLaMA2-70B.
Grok-1.5
xAI's Grok-1.5 is the successor to the open-weight Grok-1, emphasizing long-context understanding and substantially stronger math, code, and reasoning performance.
SEEDS
Google's Scalable Ensemble Envelope Diffusion Sampler (SEEDS) uses diffusion models to generate very large, physically plausible weather-forecast ensembles conditioned on only one or two operational forecasts.
LLMs on University-Level Physics Coding
A controlled study pits ChatGPT variants against University of Durham physics students on Python coding assignments, finding that humans still outperform even the strongest prompt-engineered GPT-4.
Mini-Gemini
Mini-Gemini enhances vision-language models by adding a second high-resolution visual encoder that refines details without increasing the number of visual tokens consumed by the LLM.
Long-form factuality in LLMs
Google DeepMind introduces LongFact and SAFE, a prompt set and automated evaluator for judging whether the long-form answers of modern LLMs are actually factual.
Agent Lumos
Lumos is a unified recipe for training open-source LLM agents that separates high-level planning from low-level grounding so each module can be supervised and improved independently.
AIOS
AIOS treats the LLM as the "brain" of an operating-system kernel for agents, providing scheduling, memory, storage, tool, and access-control services so agent apps can share resources safely.
FollowIR
FollowIR is both a benchmark and a training set for teaching retrieval models to follow real-world, instruction-style queries rather than just match keywords.
LLM2LLM
LLM2LLM is an iterative data augmentation scheme where a strong teacher LLM generates new training examples targeted at the specific mistakes a student model makes during fine-tuning.
Grok-1
xAI open-sources Grok-1, a 314B-parameter Mixture-of-Experts base model, making it the largest openly released LLM at the time of publication.
Evolutionary Model Merge
Sakana AI proposes using evolutionary algorithms to automatically discover effective merges of open-source models, producing strong composite models without any additional training.
TacticAI
Google DeepMind, in collaboration with Liverpool FC, releases TacticAI, a geometric deep-learning system that analyzes football corner kicks and suggests alternative tactics for coaches to explore.
What Are Tools Anyway? A Survey of Tool Use in LLMs
This survey establishes a formal definition of tools as "external programs used by LMs" and systematizes when, why, and how tool-use improves LLM performance.
RankPrompt: Step-by-Step Comparisons Make LLMs Better Reasoners
RankPrompt is a prompting method that lets an LLM self-rank its own candidate answers via chains of pairwise comparisons, without needing an external verifier or additional fine-tuning.
LLM4Decompile
LLM4Decompile is the first open-source family of LLMs specialized for decompiling machine code back into readable, re-executable C source.
Agent-FLAN
Agent-FLAN redesigns fine-tuning data so that open models can learn agentic skills without sacrificing general capability, hitting new open-source SoTA for Llama2-7B-based agents.
Logits of API-Protected LLMs Leak Proprietary Information
The paper shows that the softmax bottleneck in modern LLMs means even logit-level APIs leak enough information to reconstruct hidden architectural details.