Dynamic Chunking for End-to-End Hierarchical Sequence Modeling

Learned, Not Handcrafted, Tokenization – H-Net replaces the traditional, static tokenization process with a dynamic chunking (DC) mechanism. This allows the model to learn content- and context-dependent segmentation strategies directly from data, a significant step towards true end-to-end learning.

Hierarchical Processing for Efficiency and Power – The architecture is hierarchical, resembling a U-Net. It processes raw data with a small encoder, compresses it into meaningful chunks for a larger main network to process, and then decompresses it with a decoder. This allows for efficient processing of long sequences and enables the model to learn multiple levels of abstraction.

Superior Performance and Scaling – When matched for compute and data, a single-stage H-Net operating on bytes outperforms a strong Transformer language model using BPE tokens. A two-stage H-Net scales even better, matching the performance of a token-based Transformer twice its size.

Robustness and Generalizability – H-Nets demonstrate significantly better character-level robustness and show impressive performance on languages and modalities with weak tokenization heuristics, such as Chinese, code, and DNA, showcasing the potential of true end-to-end models.