Beyond Endpoint Scores: Time- and Capacity-Conditioned Evaluation of Continual Knowledge Updating
Heejin Choi at Yonsei University shows that the ranking of continual knowledge-updating methods reverses depending on when you evaluate and how much adapter capacity the baseline gets.
Ask this paper
The reversal is large and specific. On Qwen2.5-1.5B a periodic hierarchy holds a 5.0-point advantage over rank-8 replay and an 11.6-point deficit against rank-72 replay. The method did not change; the baseline's LoRA rank did.
Timing matters independently of capacity. At high ranks, evaluating at a consolidation-aligned endpoint can suggest a tie while time-averaged replay leads by 9 to 13 points.
The pattern replicates on Llama-3.2-1B and on held-out paraphrases, so it is not a single-model artifact.
The proposed protocol is to report trajectories and capacity sweeps, and to declare a winner only when the ordering holds across the whole evaluation region; otherwise report winner regions and retention-stability-cost frontiers.
Restated conclusion for the method under test: the periodic hierarchy is a lower-update-cost operating point rather than a higher-quality one.
Abstract
Continual knowledge-updating methods are often declared superior from one final checkpoint and one conventional adapter rank. We show that this can be insufficient to identify the better operating point. Holding a periodic hierarchy fixed, we compare it with cumulative replay over a 24-month Wikidata stream while varying evaluation month, replay LoRA rank, and query formulation. The apparent winner changes across this region: on Qwen2.5-1.5B, the hierarchy's 5.0-point advantage over rank-8 replay becomes an 11.6-point deficit against rank-72 replay, and at high ranks a consolidation-aligned endpoint can suggest a tie while time-averaged replay leads by 9-13 points. The same rank-conditioned reversal appears on Llama-3.2-1B and held-out paraphrases. These results show that method ranking in continual updating can depend jointly on when performance is measured and how much replay-side adaptation capacity the baseline receives. We therefore propose reporting trajectories and capacity sweeps, and declaring a robust winner only when the ordering is stable across the evaluation region; otherwise, comparisons should report winner regions and retention-stability-cost frontiers. Under this protocol, the periodic hierarchy is a lower-update-cost operating point, not a quality winner.