Hints Help But Do They Teach? Evaluating Skills Transfer in Code Generation
Will Badr asks whether a hint that turns a failing program into a passing one supplies missing information or merely steers the model to a solution it could already reach, and finds mostly the latter.
Ask this paper
A clean control. Relevant hints, unrelated hints, and plain repeated sampling are compared on HumanEval+ and MBPP+ with executable evaluation, which is what makes the steering hypothesis testable.
The numbers point one way. For Qwen2.5-3B-Instruct, relevant hints rescue 36 of 79 failures and an unrelated hint rescues 19, while eight unhinted samples solve 46 and recover 31 of those 36. Phi-3.5-mini repeats the pattern: 42 of 101 rescued, 17 by an unrelated hint, and unhinted sampling solves 57 including 36 of the 42.
Mechanistic tests agree. A stable activation direction is shared by relevant and unrelated hints. Persistently adding it yields 14 rescues against 18 regressions with no detectable net gain, and learned low-rank interventions are positive but imprecise.
Where real information does help. Full textual specifications solve 22 of 24 context-defined problems against 5-11 for tested virtual-KV prefixes, which separates genuine specification from steering.
The author flags the confound himself. Hint conditions use different attempt budgets, so the comparisons do not isolate a purely semantic effect. That caveat is worth carrying forward into any claim that a skill file taught a model something.
Abstract
When a hint turns a failing generated program into a passing one, does it provide missing information or merely steer the model toward a solution it could already produce? We test these hypotheses on HumanEval+ and MBPP+ using executable evaluation. For Qwen2.5-3B-Instruct, adaptive relevant hints rescue 36 of 79 selected failures; an unrelated hint rescues 19, while eight unhinted samples solve 46 and recover 31 of the 36 relevant-hint rescues. Phi-3.5-mini shows the same pattern: relevant hints rescue 42 of 101 failures, an unrelated hint rescues 17, and unhinted sampling solves 57, including 36 of the 42 relevant-hint rescues. Because the hint conditions use different attempt budgets, these comparisons do not isolate a purely semantic effect. Mechanistic tests on Qwen identify a stable activation direction shared by relevant and unrelated hints. Persistently adding this direction yields 14 rescues and 18 regressions, with no detectable net accuracy gain; learned low-rank interventions have a positive but imprecise estimated effect. Full textual specifications solve 22 of 24 context-defined problems, versus 5-11 for tested virtual-KV prefixes. Post-generation hidden-state probes transfer across benchmarks, with pooled AUROC 0.806 and 0.780, but their top-one selection advantage over token confidence is statistically unresolved. Overall, relevant hints can rescue failures, but most rescued solutions are already reachable through ordinary sampling, and the internal interventions tested here do not establish task-general capability transfer.