Data

Data Management for LLMs

First page

Paper summary

A survey of data-management research for LLM pretraining and supervised fine-tuning stages.

Ask this paper

Key points

Pretraining data: Covers data quantity, quality filtering, deduplication, domain composition, and curriculum strategies for large-scale pretraining.

SFT data: Reviews instruction-data generation, quality filtering, diversity metrics, and the emerging literature on "less is more" for SFT.

Domain and task composition: Examines how task mixing affects generalization vs. specialization in fine-tuning.

Open challenges: Identifies dataset contamination, deduplication at trillion-token scale, and reproducible data recipes as the top open problems.

Every Monday