Where vs What: Decomposing Structural and Content Failures in LLM-Generated Structured Outputs
Yiwei Zhang, Chengke Wu, Li Wang and Jianqiang Li split structured-output failures into placement errors and value errors and find that structure breaks down well before content does.
Ask this paper
Two error modes conflated by every existing metric: A placement error puts the correct value in the wrong position; a value error puts a wrong value in the right position. Monolithic scoring cannot tell them apart, which hides where the model is actually failing.
Structure degrades first and faster: Across nested JSON and table tasks on six models from 7B to frontier, structural fidelity falls off earlier and more sharply than content accuracy as complexity rises.
The numbers are stark at high complexity: DeepSeek-V4-Flash with reasoning still misplaces 35% of recalled values, and Qwen2.5-7B misplaces 74%. The model retrieved the right value and put it in the wrong slot.
Ablations point at semantic shortcuts: The pattern is associated with reliance on semantic cues rather than topological understanding of the target schema, which explains why more reasoning does not close the gap.
Abstract
Structured outputs such as JSON and tables are central to modern LLM-based systems, yet generation failures are evaluated monolithically, conflating two distinct error modes: placement errors (correct values at wrong positions) and value errors (wrong values at intended positions). We introduce Structure-Content Decomposition (SCD), a framework that independently measures structural fidelity and content accuracy. Applying SCD to nested JSON and table tasks across six models (7B to frontier), we uncover a consistent phenomenon: structural fidelity degrades earlier and more sharply than content accuracy as complexity increases. At the highest complexity, even DeepSeek-V4-Flash (with reasoning) misplaces 35% of recalled values, while Qwen2.5-7B misplaces 74%. Controlled ablations suggest that this pattern is associated with reliance on semantic shortcuts rather than topological understanding of output structure. Based on these findings, we propose SA-RLVR, converting SCD metrics into verifiable rewards for reinforcement learning via GRPO. SA-RLVR successfully optimizes structural addressing across distinct topologies: it lifts JSON Value Placement Accuracy (VPA) from 26% to 63% while generalizing to held-out schemas; moreover, it consistently drives VPA improvements in the table domain, demonstrating that structure-aware rewards can directly enhance multi-domain structural positioning.