Aligning Vision Models with Human Perception

Google DeepMind presents a method to align AI vision models with human visual understanding by addressing systematic differences in how models organize visual representations, demonstrating that alignment improves robustness, generalization, and reliability across diverse vision tasks. - Odd-one-out reveals misalignment: Using classic cognitive science tasks, researchers found vision models focus on superficial features like background color and texture rather than high-level semantic concepts humans prioritize. - Three-step alignment method: A frozen pretrained model trains a small adapter on the THINGS dataset, creating a teacher model that generates human-like judgments. This teacher creates AligNet, a massive dataset of millions of odd-one-out decisions, then student models are fine-tuned to restructure their internal representations. - Representations reorganize hierarchically: During alignment, model representations move according to human category structure, with similar items moving closer together while dissimilar pairs move further apart. This reorganization follows hierarchical human knowledge without explicit supervision. - Improved performance across tasks: Aligned models show dramatically better agreement with human judgments on cognitive science benchmarks and outperform originals on few-shot learning and distribution shift robustness.