Survey: Text Data Augmentation for LLMs

Classifies augmentation methods – It defines four categories: (1) Simple augmentation – basic text manipulations like synonym replacement, cropping, etc.; (2) Prompt-based augmentation – using an LLM with specific prompts to generate new training examples (taking advantage of the LLM’s own generative power; (3) Retrieval-based augmentation – pulling in external knowledge or contexts (via search or databases) to ground the generated text in facts; and (4) Hybrid augmentation – combinations of the above, or multi-step strategies.

LLMs as data generators – A key insight is that modern LLMs can create high-quality synthetic data to improve themselves. By carefully prompting an LLM to produce variations of a task (for example, ask ChatGPT to come up with new math word problems), one can dramatically expand a training set. The survey discusses prompt design for this purpose and how to ensure the generated data is diverse and useful.

Post-processing and filtering – Augmented data isn’t always perfect. The survey covers techniques to refine and filter generated data. For instance, verifying facts with a secondary model or removing examples that might introduce errors. This step is crucial to prevent “garbage in, garbage” out when augmenting data.

Evaluation and future directions – It outlines common tasks where data augmentation is used (like low-resource language translation, QA, etc.) and how to evaluate the impact (improvement in accuracy, robustness, etc.). Finally, it discusses challenges (e.g. ensuring augmentation doesn’t distort data distribution, avoiding model bias reinforcement) and opportunities for new research.