CM3Leon

Autoregressive multi-modal: Unifies text and image generation in a single autoregressive token-based architecture, handling both modalities in any order.

5x training efficiency: Achieves SOTA image generation quality with 5x less training compute than comparable methods due to retrieval augmentation and instruction tuning.

Instruction tuning for images: Demonstrates that supervised fine-tuning and instruction tuning - originally developed for LLMs - also massively improves multimodal generation quality.

Any-to-any direction: Early proof-of-concept for unified any-to-any multi-modal models, pre-dating and inspiring 2024 systems like Chameleon and GPT-4o.