Data · Training · Reasoning

Llemma

Free while signed in. Answers cite the passages they came from.

First page

The curator’s take

Llemma is an open LLM for mathematics built via continued pretraining of Code Llama on the Proof-Pile-2 dataset.

Key points

Proof-Pile-2 dataset: Mixes scientific papers, math-heavy web pages, and mathematical code into a focused math-pretraining corpus.

Code Llama base: Uses Code Llama as the base model, leveraging its existing code proficiency as a scaffold for formal-style math reasoning.

Beats unreleased Minerva: Outperforms open base models and the unreleased Minerva on the MATH benchmark at comparable scale.

Full open release: Releases model, dataset, and code - positioning Llemma as a reproducible starting point for open mathematical LLM research.

Every Monday

The same picks and the same summaries, in your inbox. Free, and 176 issues deep.