Disentangling the Factors of Convergence between Brains and Computer Vision Models
Large self-supervised ViTs trained on natural images develop brain-like internal representations. This paper teases apart what drives that convergence by varying model size, training amount, and image type in DINOv3, then comparing model activations to human fMRI (space) and MEG (time) with three metrics: overall linear predictability (encoding), cortical topography (spatial), and temporal alignment (temporal). Result: all three factors matter, and alignment unfolds in a consistent order from early sensory to higher associative cortex.
Ask this paper
Setup and metrics: Eight DINOv3 variants spanning sizes and datasets; comparisons use encoding, spatial, and temporal scores against NSD fMRI and THINGS-MEG.
Baseline alignment: fMRI predictability concentrates along the visual pathway (voxel peaks around R≈0.45). MEG predictability rises ~70 ms after image onset and remains above chance up to 3 s. Spatial hierarchy holds (lower layers ↔ early visual; higher layers ↔ prefrontal; r≈0.38). Temporal ordering is strong (earlier MEG windows ↔ early layers; r≈0.96).
Training dynamics: Alignment emerges quickly but not uniformly: temporal score reaches half its final value first (~0.7% of training), then encoding (~2%), then spatial (~4%). Early visual ROIs and early MEG windows converge sooner than prefrontal ROIs and late windows (distance-to-V1 vs half-time r≈0.91; time-window vs half-time r≈0.84).
Scale and data effects: Bigger models finish with higher encoding, spatial, and temporal scores; gains are largest in higher-level ROIs (e.g., BA44, IFS). Human-centric images beat satellite and cellular images across metrics and ROIs at matched data volume.
Cortical correlates: ROIs whose model alignment appears later are those with greater developmental expansion, thicker cortex, slower intrinsic timescales, and lower myelin (e.g., correlations up to