Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models
Kalyani Marathe, Artidoro Pagnoni and colleagues from the University of Washington and Meta FAIR, including Mike Lewis, Luke Zettlemoyer and Srinivasan Iyer, compare distilled byte-level and token-level 1B models trained on up to 1 trillion bytes.
Ask this paper
Logit conversion: Two methods convert a token teacher's logits into byte logits, an approximate one (Marginalize-It) and an exact one (End-Of-Token).
Crossover: Token models lead at low compute but plateau, while byte models start worse and overtake them with more compute, reaching a higher ceiling.
Asymptote: Extrapolated scaling laws predict distilled End-Of-Token-1B beats distilled Token-1B by up to 4%, and Llama 3.2-1B, Gemma-3-1B-pt and Gemma 2B by up to 6.5%, 8.1% and 2.1%.
Data efficiency: Distilled byte models match distilled Token-1B with one-sixth of the training data.
Storage: A 256-byte vocabulary removes the need for top-k truncation when dumping teacher logits and cuts logit storage to about one-fifth.
Abstract
Small models are made more capable through distillation from a larger one that shares their tokenization scheme. However, do distilled byte and token models behave similarly in terms of scaling trends as compute and data increases? To enable this comparison, we introduce two variants to efficiently convert token logits to Byte Logits: 1) approximate: Marginalize-It, and 2) exact: End-Of-Token. We then present the first large scale study of overtraining decoder-only dense transformer models varying two dimensions simultaneously: the tokenization scheme (Tokens, Bytes, Bytes w/ eot) and the training objective (Distillation vs. Cross-Entropy), sweeping layer-parameter-matched models with roughly 1 billion parameters up to 1 trillion bytes of data. Across eight benchmarks spanning three categories: Multiple Choice QA, Language Generation, and Machine Translation, we find that Token-1B models outperform byte models (End-Of-Token-1B and Bytes-1B) in the low-FLOP regime but eventually plateau; byte models start worse yet surpass Token-1B models with more compute, reaching a higher downstream task performance ceiling. Extrapolating the average top-1 error vs. validation BPB scaling laws predicts that, asymptotically, distilled End-Of-Token-1B outperforms distilled Token-1B by up to 4%. They are also far more data efficient, matching the performance of distilled Token-1B using only one-sixth of the training data. Moreover, by operating over a small vocabulary of 256 bytes instead of on the order of 100K tokens, they circumvent the need for top-k truncation during logit dumping, while also reducing logit storage costs to roughly one-fifth. Finally, our downstream performance scaling laws predict that our distilled End-Of-Token-1B models asymptotically surpass the Llama 3.2-1B, Gemma-3-1B-pt, and Gemma 2B models on averaged downstream tasks by up to 6.5%, 8.1%, and 2.1%, respectively.