Many-shot Jailbreaking
Anthropic shows that long-context windows enable a new attack where hundreds of fake user/assistant dialogues are packed into a single prompt, coaxing frontier LLMs to answer the final harmful question despite safety training.
SWE-Agent
Princeton's SWE-agent pairs a language model with a custom agent-computer interface (ACI) that exposes file navigation, editing, and test execution as discrete tools, letting the agent autonomously resolve real GitHub issues.
Mixture-of-Depths
DeepMind proposes dynamically allocating transformer FLOPs across sequence positions via a top-k router, so "easy" tokens skip expensive blocks while "hard" tokens get full computation.
Long-context LLMs Struggle with Long In-Context Learning
LongICLBench stress-tests 13 long-context LLMs on extreme-label classification with up to 174 classes and 50K-token prompts, exposing sharp quality cliffs beyond 20K tokens.
Visualization-of-Thought
Microsoft's Visualization-of-Thought (VoT) prompts LLMs to emit intermediate "mental images" of their reasoning state, lifting spatial-reasoning accuracy on grid-world tasks and beating multimodal baselines that actually see images.
The Unreasonable Ineffectiveness of the Deeper Layers
The paper shows that open-weight LLMs tolerate removing up to half of their transformer blocks with only minor degradation, provided a short QLoRA pass is used to heal the damage afterwards.
JetMoE
MyShell's JetMoE-8B is an open MoE model trained for under $100K that matches or beats LLaMA2-7B, showing that competitive LLM training can be achieved on modest budgets with public data.
ReFT: Representation Finetuning for LMs
Stanford's ReFT freezes the base model and instead learns small interventions on hidden representations at selected layers, offering a more parameter-efficient alternative to LoRA-style PEFT.
Advancing LLM Reasoning (Eurus)
OpenBMB's Eurus is a suite of reasoning-specialized LLMs (7B and 70B) fine-tuned on UltraInteract, a new alignment dataset built around preference trees for complex math, code, and logical tasks.
Training LLMs over Neurally Compressed Text
The paper proposes Equal-Info Windows, a neural compression scheme that segments text into equal-bit-length blocks so an LLM can train directly on compressed bytes without losing learnability.