← All papers  /  Nov 16, 2026
Safety

Interpretable weight-space programming builds transparent transformers and brain encoding models

First page
Interpretable weight-space programming builds transparent transformers and brain encoding models
The curator’s take

Yash Mehta, Chandan Singh, Jianfeng Gao and colleagues at Microsoft Research, Johns Hopkins, Columbia and NYU introduce interpretable weight-space programming (iWSP), in which an LLM agent writes the weights and features of a model directly in code, without backpropagation, to build networks a scientist can read.

Ask this paper

Key points
01

Method. The agent iterates 30 to 50 times, hypothesizing computations, writing them as explicit weights and features, testing, and revising. LLM audits check that the resulting models are interpretable.

02

Transformers by hand. All 26 runs on algorithmic tasks reach 100% accuracy on visible and hidden sets, often by implementing a known algorithm in the weights. The smallest 100%-accurate model has a median of 166 parameters with Claude Opus 5.5, versus 41,600 for Sonnet 5.5 and 100,000 for Haiku 5.5.

03

NLP is harder. On NLP tasks models plateau near 190,000 parameters and paraphrase stays below its baseline, so the hand-built models are interpretable but less accurate.

04

Neuroscience. In visual cortex, iWSP outpredicts existing interpretable baselines and generalizes to unseen data, with a final 865-line program producing 9,996 features from 14 explicit computations. In language fMRI it surfaced a confound in the data, and in auditory ECoG it found acoustic features that predict neural activity.

Abstract

Neural networks are powerful predictors, yet their opacity limits their usefulness in science, where the goal is explanation rather than mere prediction. In neuroscience, for example, deep encoding models accurately predict brain activity in response to natural stimuli, but struggle to reveal which stimulus features drive those predictions. Here we introduce interpretable weight-space programming (iWSP), a method in which an LLM agent iteratively builds interpretable models by explicitly programming their weights and features in code, without gradient-based training (backpropagation). Without backpropagation to tune parameters, the agent must draw on its prior knowledge and reason over previous trials to hypothesize, hand-craft, and refine human-interpretable computations mirroring the workflow of a human scientist. LLM audits confirm the interpretability of the hand-crafted models. We first validate iWSP by tasking it with constructing transformers for canonical benchmarks such as arithmetic and sentiment classification, finding that LLM agents can successfully craft compact, accurate networks directly. Next, we apply iWSP to build neural encoding models across vision, language, and audition. In the visual cortex, iWSP outpredicts existing interpretable baselines, uncovers distinct computations across early/higher visual areas, and generalizes to unseen data. In language fMRI, iWSP surfaces a confound in the data and in auditory ECoG, it systematically discovers interpretable acoustic features that predict neural activity. Together, these results establish iWSP as a general paradigm for aligning high predictive performance with human-interpretable scientific discovery.

Every Monday
Get next week’s papers.
Subscribe on Substack