MEGABYTE

Two-level architecture: Combines a large global Transformer over patches with a smaller local Transformer over bytes within each patch.

Sub-quadratic attention: Achieves sub-quadratic self-attention cost through the patch-level hierarchy, enabling million-byte sequences.

Decoding parallelism: Improves decoding parallelism compared to flat Transformers that must decode token-by-token.

Tokenization-free: Operates directly on bytes without tokenizers - potentially avoiding tokenizer failure modes.