Agentic Knowledge-Model Co-Evolution via Consistency Regularization
Sirui Chen, Jingrui He and Zheng Li (UIUC, Georgia Tech, Amazon) introduce SPLICE, which keeps only the knowledge extracted from agent experience whose utility gain holds across several different ways of analyzing the same experience, and interleaves that with model training.
Ask this paper
Problem. Different ways of analyzing the same trajectories produce skills and memories with very different usefulness, even when each looks reasonable.
Method. SPLICE perturbs how experience is analyzed to produce several knowledge candidates, and updates the knowledge store by the common gain, which combines utility improvement with its consistency across these views.
Co-evolution. During model updates, self-distillation internalizes the evolved knowledge alongside RL, and knowledge with consistent utility loss serves as a contrastive reference. Knowledge and model updates alternate.
Result. Across domains SPLICE beats knowledge-evolution and model-training baselines by up to 7.1 and 4.6 points in average accuracy, cuts output tokens per correct solution by up to 44%, and its knowledge transfers across model sizes and families.
Abstract
LLM agents can evolve by learning from accumulated experience. Existing approaches extract reusable external knowledge, such as skills and memories, from experience to guide subsequent task solving, with some further internalizing this knowledge through model training. However, different ways of analyzing and transforming the same experience can yield knowledge with substantially different utility, even when the resulting guidance appears reasonable. Inspired by consistency regularization, we identify consistent utility gains across different ways of analyzing the same experience, rather than relying on any single way of analyzing it. We introduce SPLICE, a framework for consistency-regularized knowledge-model co-evolution. It constructs evolution views by perturbing how the same experience is analyzed, producing different knowledge candidates. Knowledge updates are then guided by common gain, which accounts for utility gains and their consistency across views. During model evolution, self-distillation internalizes the evolved knowledge alongside reinforcement learning, using knowledge associated with consistent utility degradation as a contrastive reference. The two processes interleave, with model updates driving further knowledge evolution. Experiments across diverse domains show that SPLICE outperforms knowledge evolution and model training baselines by up to 7.1 and 4.6 points in average accuracy, while its evolved knowledge reduces output tokens per correct solution by up to 44% and transfers across model sizes and families.