Skill-based Agentic Evaluation for Real-time Data Science Tasks
Aniruddha Tamhane and colleagues at Adobe evaluate data-science agents on live data by writing each reference answer as executable code and scoring responses at the level of atomic facts.
Ask this paper
Ground truth as code: Each expected answer is a Python function run against the live system at evaluation time, so the reference tracks data drift and fails loudly when an upstream API changes.
Factoid scoring: An LLM judge splits both the agent response and the computed answer into atomic claims and scores precision, recall and accuracy regardless of format.
Agreement: Against expert labels, ground-truth-as-code raises MCC from 0.331 to 0.427 and cuts token cost per case by 16%; a self-directed judge without ground truth is anti-correlated (MCC -0.379).
Packaging: The pipeline runs as a harness skill; the same model (Claude Sonnet 4.6) serves as skill and judge, which the authors list as a limitation.
Abstract
We present a framework for evaluating data-science agents on live, continuously updated data using executable ground truth and format-agnostic factoid scoring. Consider this example query: "what were last week's audience sizes"---the reference answer changes as the underlying data changes, so static references become outdated and standard LLM-as-a-judge pipelines cannot verify responses against a fixed ground truth. Our central contribution, ground-truth-as-code, encodes each expected answer as an executable reference function that recomputes the answer directly from live data at evaluation time, ensuring the reference remains consistent with the system it describes. We combine this with a factoid-level, format-agnostic judge that decomposes both the agent's response and the computed ground truth into atomic claims and scores precision, recall, and accuracy over them, irrespective of the response format (prose, list, table, HTML, etc.). The approach is applicable to agents whose expected outputs can be expressed as executable data computations. We validate the framework through a human--LLM agreement study on an internally developed machine learning skill deployed in production, using a synthetic database constructed to reproduce production schemas and entity relationships. Relative to a natural-language ground-truth baseline, our method achieves a 29% improvement in the Matthews Correlation Coefficient (MCC)---a class-balanced measure of agreement between expert annotators and LLM-as-a-judge predictions---and a 16% reduction in token consumption per test case, while a self-directed baseline lacking explicit ground truth is anti-correlated with human judgment. Agents that perform multi-source data integration and computation over non-stationary data are routinely deployed in industry; we propose ground-truth-as-code as a practical methodology for their evaluation.