FlexComp: One Model for Every Ratio in Context Compression
Kaiyan Zhao, Zhongtao Miao, Akiko Aizawa and Yoshimasa Tsuruoka (The University of Tokyo and National Institute of Informatics) train one soft context compressor that works at any compression ratio and choose the ratio for each input, instead of training a separate model for every fixed ratio.
Ask this paper
Problem: Soft compressors turn a context into a few memory tokens that a frozen LLM reads, but each deployed ratio needs its own trained model and the same ratio is applied to every input.
Matryoshka training: The memory budget K is sampled per training instance, so a single model learns to compress at any ratio. The method works with ICAE, 500xCompressor and SAC.
Per-input budget: The budget is chosen either by a confidence-based cascade that tries tighter budgets first, or by a lightweight learned K predictor that picks it in one pass.
Results: On MRQA the cascade keeps over 98% of the mildest ratio's accuracy at up to 266x average compression. The predictor reaches 158 to 236x within 0.7 F1 of the mildest ratio, and at serving batch sizes it cuts context KV cache by 50% and raises decoding throughput by 47%.
Abstract
Soft context compression condenses a context into a few memory tokens that a frozen LLM consumes in place of the raw text, but existing compressors fix the compression ratio at training and inference: each deployed ratio requires a separately trained model, and the chosen ratio is applied uniformly to all inputs, whose actual needs vary drastically. We propose FlexComp, a method-agnostic framework that decouples the ratio from both training and deployment: Matryoshka-style training samples the memory budget $K$ per instance, turning one model into an any-ratio compressor, and the budget is then chosen per input by: (1) confidence-based cascade routing or (2) a lightweight learned $K$ predictor. Across ICAE, 500xCompressor, and SAC on MRQA, a single FlexComp model matches separately trained fixed-ratio specialists with minimal degradation. Cascade routing preserves over 98% of the mildest ratio's accuracy at up to 266x average compression; the $K$ predictor, in a single compression-decoding pass, reaches 158-236x within 0.7 F1 of the mildest ratio. At serving-scale batch sizes, the $K$ predictor cuts context KV cache by 50% and improves decoding throughput by 47%.