Same Context, Different Agents: A Behavioral Study of Semantic Layers, Orchestration and Compaction in Data Agents
Zhivar Sourati, Marianne Menglin Liu and colleagues (USC, Oracle AI and UPenn, with Dan Roth) test three context-engineering techniques for data agents with seven LLMs on 1,514 long-horizon tasks and find that their effect depends on how the agent already behaves.
Ask this paper
Three behavior groups. Without any technique, agents act as investigators, as premature answerers that answer without inspecting the data, or as early stoppers that give up.
Fit determines the gain. Each technique produces its intended effect, from more investigation to smaller prompts, but accuracy improves only for the group it fits, for example premature answerers under orchestration.
Investigators get worse. A semantic layer stops them from inspecting data, orchestration makes them overconfident and costs 5 points, and compaction makes them repeat steps.
Combinations. Using techniques together gives smaller gains and smaller losses than the sum of their separate effects. The paper gives guidance for choosing techniques by agent behavior, task demands and cost.
Abstract
Answering questions over heterogeneous data means combining evidence from files, databases, documents and the web. Data agents automate this, but their success depends on context. Three context-engineering techniques (CETs) help by shaping it: semantic layers describe the data, orchestration splits work among agents, and compaction condenses interaction history. Existing studies validate each CET alone, leaving open how agent behavior shapes its effect. We present a comprehensive analysis of the three CETs, alone and combined, with seven large language models (LLMs) on 1,514 long-horizon tasks from five benchmarks, covering behavior, accuracy and cost. We find that data agents without CETs form three behavior groups: investigators, premature answerers that answer without inspecting data, and early stoppers that give up. Each CET delivers its known benefit, from more investigation to smaller prompts, but only the behavior group it matches turns that benefit into accuracy, such as premature answerers under orchestration. For investigators, the same CETs can hurt: the semantic layer stops them from looking, orchestration makes them overconfident, costing 5 percentage points, and compaction makes them repeat steps. Combining CETs yields smaller gains and losses than their sum. We derive guidance for choosing CETs by agent behavior, task demands and accuracy-cost trade-offs.