A Heart Means Recursion: Exploring Visual Meaning and Communication in VLMs
Alexa Tartaglini, Zhengxuan Wu, Thaddäus Wiedemer, Martin Wattenberg, Been Kim and colleagues (Google DeepMind, Stanford) report that simple images such as a red heart can change how VLMs solve unrelated coding problems, and that models cannot explain this effect when asked.
Ask this paper
Visual steering. On inorder binary tree traversal, which has equally valid recursive and iterative solutions, a red heart in context moves Claude Opus 4.7 and GPT-5.4 to recursion in 100% of samples. A yellow checkmark moves Gemini 3.1 Pro to 90%. These effects match or approach directly instructing the model to use recursion.
Concept, not pixels. Text captions of the same symbol do not steer equivalently, random or naturalistic images steer far less, and the effect depends on shape orientation and arrangement. Steering also carries over to non-coding tasks.
Unfaithful explanations. Models' direct interpretations of the symbols and their thinking traces diverge from their behavior. For GPT-5.4, 74.7% of traces attribute the choice incorrectly.
Model-to-model communication. Models send shape primitives that they and humans find intuitive, but these often fail to steer other models. In games with abstract concepts and stories, models do transmit meaning visually, sometimes in ways humans find non-intuitive.
Abstract
Do vision-language models (VLMs) share our ability to communicate visually? We share preliminary findings, starting from a surprising observation: simple visual symbols like a red heart can strongly influence model behavior on seemingly unrelated tasks, such as steering models towards recursive solutions on simple coding problems. Notably, this steering effect is largely implicit, as models' direct interpretations and thinking traces about visual symbols diverge substantially from their observed behavior. In visual communication between models, this leads to a generation--interpretation gap: while models tend to send shape primitives that they (and humans) find intuitive, these visual cues often fail to steer other models. In further communication games involving abstract concepts and stories, models can creatively use visuals to transmit meaning to one another, although some communications remain non-intuitive to humans. These findings reveal that VLMs might be capable of rich, creative, and visually-grounded communication, but their interpretation and production of visual meaning diverges from human conventions in surprising ways. Together this work highlights both the promise and challenges of visual communication between humans and VLMs.