Published Unlearning Numbers Move Per Checkpoint, and Not Because the Removed Data Survives: An Audit of 263 Released Batch-Normalized Checkpoints
Junlong Shen and Xingyu Li (University of Alberta) audit released batch-normalized unlearning checkpoints and show that published unlearning numbers change when the batch-norm statistics are refit on kept data, with the weights left bit-identical.
Ask this paper
Undocumented state: Unlearned models and their retrained references both ship batch-normalization statistics that no gradient step wrote and no release documents.
Refit effect: Refitting those statistics on kept data at bit-identical weights moves 47 of 221 checkpoints beyond the seed spread of their own release, including cases where the method's average does not move.
Cause: Swapping kept records for removed ones in the fitting pool barely changes results, while the distance between shipped statistics and any refit tracks the change. The removed data is not surviving in the statistics.
Consequence: Twelve published verdicts cross, four clear a measured recalibration budget, and two clear it on every replicate. Releases should state the fitting convention next to each number.
Abstract
An unlearning audit reads its verdict off numbers that an unlearned model and its retrained reference each publish, and both also ship batch-normalization statistics that no gradient step wrote and no release records. Refitting them on kept data at bit-identical weights moves 47 of 221 released checkpoints past the spread their own release's seeds show, several inside a method whose average does not move: what moves is the checkpoint's property, not its method's. What does the moving is not the removed data surviving in the state: exchanging kept records for removed ones inside a fixed fitting pool moves a published cell by almost nothing, while how far a checkpoint's shipped state has drifted from any refit does track it. The consequence for a published decision is real but narrow: twelve verdicts cross, four clear a measured recalibration budget, two clear it on every replicate, and a population we trained and sited near its own criterion yields none. A release should therefore name the fitting convention beside the number, on the batch-normalized vision models where this channel exists.