Compute Optimal Scaling of Skills
Free while signed in. Answers cite the passages they came from.
Researchers from the University of Wisconsin and Meta AI investigate how different skills (knowledge-based QA vs. code generation) exhibit contrasting optimal scaling behaviors in LLMs. Their key question: does the compute-optimal trade-off between model size and data volume depend on the type of skill being learned? Surprisingly, the answer is yes—they show distinct “data-hungry” vs. “capacity-hungry” preferences per skill. Highlights:
Skill-dependent scaling laws – Traditional scaling laws optimize the overall loss on a generic validation set. However, this paper shows that knowledge tasks prefer bigger models (capacity-hungry), while code tasks prefer more data tokens (data-hungry).
Differences persist even after balancing data – Tweaking the pretraining mix (e.g. adding more code data) can shift that skill’s optimal ratio, but fundamental differences remain. Knowledge-based QA still tends to need more parameters, code still benefits from bigger data budgets.
Huge impact of validation set – Choosing a validation set that doesn’t reflect the final skill mix can lead to misaligned compute-optimal model sizes by 30%–50% at lower compute scales. Even at higher scales, suboptimal validation sets skew the best parameter count by over 10%.
Practical takeaway – Model developers must pick or design validation sets that represent the real skill mix. If your ultimate goal is to excel at knowledge-based QA, you likely need a more capacity-hungry strategy. If it’s coding tasks, you might focus on data-hungry training.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack