Beneath the Surface of Chains-of-Thought: A Mechanistic Interpretation of Reasoning Operations in LLMs
Seogyeong Jeong and colleagues at KAIST and NAVER AI Lab test whether the functional operations inside a chain of thought, such as problem formulation, goal decomposition and deduction, have distinct geometric structure in hidden representations.
Ask this paper
Separability, and where it peaks: Operations are separable in held-out representations, with separability highest in middle layers. The authors check that this is not explained by lexical or positional confounds.
Same token, different representation: Identical surface tokens are represented differently depending on the operation of the surrounding chunk, so the structure tracks function rather than wording.
Layer trend: Token-wise operation alignment becomes more distributed over spans as depth increases, rather than staying concentrated on individual tokens.
Causal check: Attention-masking interventions show that operation-aligned representations at the start of a chunk depend on preceding reasoning context, so the operation label is constructed from the trace, not read off the current token. Code is at github.com/naver-ai/beneath-cot.
Abstract
Reasoning in large language models unfolds through diverse functional operations, such as problem formulation, goal decomposition, and deduction. Although these operations are explicitly distinguished in text, little is known about how they are geometrically organized in representation spaces. To this end, we investigate whether distinct reasoning operations exhibit corresponding geometric structure in hidden representations. We find that operations are separable in held-out representations, with separability peaking in middle layers, and verify that this structure is not explained by lexical or positional confounds. Across layers, token-wise operation-alignment becomes more distributed over spans, while identical surface tokens are represented differently depending on the operation of its surrounding chunk. Attention-masking interventions further show that operation-aligned representations at chunk onset depend on preceding reasoning context. Consequently, our work demonstrates that language models maintain representational correspondence between linguistic reasoning expressions and their internal geometric structures. Code and project materials are available at https://github.com/naver-ai/beneath-cot.