Better Language Models of Code through Self-Improvement

Self-improvement loop: Generates pseudo training data from the model's own knowledge gained through pretraining and fine-tuning.

Iterative bootstrapping: Adds the generated data to the training set for the next training iteration, creating a self-improvement loop.

Multi-framework gains: Shows consistent improvements across different code LLM frameworks on code generation tasks.

Self-improvement research: An early example of the self-improvement paradigm for LLMs that would later mature in 2024's self-rewarding and self-play approaches.