AI Papers of the Week
Every paper worth reading in AI, hand-picked one week at a time.
Discover and explore top AI papers with Claude Code or Codex
npx @dair-ai/mcp setup
Extracting Interpretable Features from Claude 3 Sonnet
presents an effective method to extract millions of abstract features from an LLM that represent specific concepts; these concepts could represent people, places, programming abstractions, emotion, and more; reports that some of the discovered features are directly related to the safety aspects of the model; finds features directly related to security vulnerabilities and backdoors in code, bias, deception, sycophancy; and dangerous/criminal content, and more; these features are also used to intuititively steer the model’s output.
Risks and Opportunities of Open-Source Generative AI
analyzes the risks and opportunities of open-source generative AI models; argues that the overall benefits of open-source generative AI outweigh its risks.
Fine-tuning and Hallucinations
studies the impact of fine-tuning on new knowledge on the hallucination tendencies of LLMs; the setup includes fine-tuning examples that include new knowledge; shows that LLMs struggle to acquire new factual knowledge via fine-tuning; also finds that as new knowledge is learned it increases the model’s tendency to hallucinate.
Inner Workings of Transformer Language Models
presents a technical introduction to current techniques used to interpret the inner workings of Transformer-based language models; it provides a detailed overview of the internal mechanisms implemented in these models.
Multimodal LLM Hallucinations
provides an overview of the recent advances in identifying, evaluating, and mitigating hallucination in multimodal LLMs; it also provides an overview of causes, evaluation benchmarks, metrics, and other strategies to deal with challenges related to detecting hallucinations.
Reducing Hallucination in Structured Outputs via RAG
This paper deploys a compact RAG pipeline - small retriever plus small LM - for an enterprise workflow-generation task and shows it reduces hallucination while improving out-of-domain generalization vs a baseline LLM.
Best Practices and Lessons on Synthetic Data
Google DeepMind's survey-style position paper on synthetic data for LLMs. It covers applications, quality-assurance principles, and the open challenges of factuality, fidelity, bias, and privacy.
Overview of Multilingual LLMs
A first-of-its-kind survey on multilingual LLMs, organized by multilingual alignment principles rather than model-family hierarchy. The authors propose a unified taxonomy and collect open resources to accelerate future research.
Many-shot Jailbreaking
Anthropic shows that long-context windows enable a new attack where hundreds of fake user/assistant dialogues are packed into a single prompt, coaxing frontier LLMs to answer the final harmful question despite safety training.
Logits of API-Protected LLMs Leak Proprietary Information
The paper shows that the softmax bottleneck in modern LLMs means even logit-level APIs leak enough information to reconstruct hidden architectural details.
Stealing Part of a Production Language Model
The paper demonstrates the first practical attack that extracts the embedding-projection layer of production LLMs through their ordinary logit APIs.
On the Societal Impact of Open Foundation Models
Stanford CRFM's policy paper proposes a rigorous framework for assessing the *marginal* risk of open-weight foundation models relative to closed models and pre-existing technologies.
Red Teaming Visual Language Models
Introduces the first dedicated red-teaming benchmark for VLMs, covering vulnerabilities unique to multimodal inputs.
Patchscopes
Patchscopes is a general framework for inspecting and intervening on LLM internals by "patching" hidden representations into a second inference pass.
Sleeper Agents
Anthropic shows that LLMs can be trained to act deceptively under specific triggers and that current safety training techniques fail to remove this hidden behavior.
TrustLLM (Trustworthiness in LLMs)
A 100+ page study that defines a principled framework for trustworthy LLMs and benchmarks 16 mainstream models across it.
Persuasive Adversarial Prompts (PAP)
Turns 40 human-persuasion techniques into a taxonomy of jailbreaks that achieve 92% attack success on frontier models without any optimization.
Adversarial Machine Learning (NIST)
NIST's official taxonomy of adversarial machine learning, intended to standardize terminology for policy and practice.
Mitigating Hallucination in LLMs
A survey cataloging 32 hallucination-mitigation techniques and organizing them into a practical taxonomy.
Exploiting Novel GPT-4 APIs
A red-team study of three newer GPT-4 API surfaces - fine-tuning, function calling, and knowledge retrieval - that reveals each introduces new attack vectors.
Fact Recalling in LLMs
A mechanistic-interpretability study showing that early MLP layers function as a lookup table for factual recall.
Adversarial Attacks on GPT-4
Demonstrates that a trivially simple random-search procedure can jailbreak GPT-4 with high reliability.
FunSearch
DeepMind's FunSearch uses LLMs as a mutation operator in an evolutionary loop to discover genuinely new mathematical knowledge.
Weak-to-Strong Generalization
OpenAI's superalignment team shows that weak supervisors can still elicit capabilities from much stronger models - a first empirical signal for scalable oversight.