Inexact Unlearning Needs More Careful Evaluations (Hayes et al.)
Why
This is the paper that decides how the thesis must measure forgetting. It shows that the usual
membership-inference checks make approximate unlearning look far more private than it is.
Summary
Bibliographic. Jamie Hayes, Ilia Shumailov, Eleni Triantafillou, Amr Khalifa, Nicolas Papernot
(Google DeepMind). Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of
Privacy. arXiv:2403.01218v3, 21 May 2024. 37 pages.
Problem
An unlearned example should be indistinguishable from an example that was never in the training
set, which is exactly the membership-inference question. The paper adapts MIAs to the unlearning
setting (“U-MIAs”) and separates them into two classes:
- Population U-MIA. One attacker is trained/instantiated for all examples (e.g. a single
classifier or a fixed loss-threshold rule). This is the commonly used U-MIA in the unlearning
literature. - Per-example U-MIA. A dedicated attacker is constructed for each example under attack. The
paper’s implementation, U-LiRA, adapts LiRA (likelihood-ratio attack) to unlearning.
Experimental setup
- Vision: ResNet-18 on CIFAR-10, with shadow models trained on random half-splits of the
training set. - Language: PaLM 2 (Bison) fine-tuned on LIMA; forget sets are PII, poison, and random data.
- Unlearning algorithms benchmarked (as listed in the paper): SCRUB, fine-tuning, NegGrad
(gradient ascent on the forget set), GradDesc (gradient descent on the retain set), NegGrad+,
Catastrophic Forgetting-k (CF-k), Exact Unlearning-k (EU-k), and -sparse MU. SISA is
not among the benchmarks. - Metric: attack accuracy over unlearning epochs, with the key comparison being U-LiRA versus
the population baseline.
Findings
- Per-example attackers are much stronger than population attackers. The commonly used
population U-MIAs overestimate the privacy protection of existing unlearning techniques, on both
vision and language models. - Vulnerability varies strongly across examples. Different examples are unlearned at
different rates; several algorithms reduce vulnerability for some examples while increasing it
for others. - The retain set can get worse. The privacy protection of the remaining training examples may
worsen as a consequence of unlearning, because the unlearning update moves the model in a way
that makes retained examples more distinguishable. - U-LiRA stays strong where population attacks say “forgotten”. In the language experiment,
U-LiRA attack accuracy does not drop below 80% except for random data; NegGrad and GradDesc fail
to unlearn any samples successfully when measured this way. - Timing matters. For naturally occurring language data (PII, poison), attack accuracy
decreases slowly with more unlearning epochs; for random data it is most successfully unlearned
after about 2 epochs and then rises again. - Naive per-example stopping criteria do not fix it. The paper tries tailoring unlearning
stopping rules to individual examples and finds it insufficient.
Why this happens
Two sources of variance undermine the “every example must be indistinguishable” requirement:
- Optimisation variance. Mini-batch order is random, so the same unlearning algorithm run twice
gives different final losses; a sample can sit on the tail of the in/out distribution and be
identifiable by chance. - Different unlearning rates. Algorithms that optimise average-case forget-set accuracy (e.g.
stopping when forget accuracy matches a validation set) are calibrated for the average, not for
each example. U-LiRA attacks each example individually, which exposes exactly this mismatch.
Limitations
The paper’s own scope is vision classifiers and a fine-tuned language model; it does not study
generative models or graphs. Shadow-model cost is significant.
Thesis hook
- The ER notebook’s MIA is a population attack: per-node NLL, thresholded, AUC computed. This
paper predicts it will overestimate forgetting. The final thesis evaluation must at least
acknowledge this, and ideally use a per-example attack. - “Retain-set privacy can worsen” is directly relevant to graph unlearning: deleting one graph (or
node set) may make the remaining graphs easier to identify. - The paper’s negative result about stopping criteria is a warning for the thesis: a fixed
”unlearning until forget accuracy ≈ chance” rule is not evidence of privacy. - It supplies the evaluation design for the eventual diffusion experiments: U-LiRA-style per-record
attacks, not a single average.
Open questions
- How do we build shadow models for graph diffusion? Retraining DiGress many times is expensive;
is a smaller surrogate or a GNN classifier enough? - What is the graph analogue of a “per-example” attack: per-graph likelihood ratio, per-edge
reconstruction, or per-subgraph MIA? - Does the retain-set degradation effect appear when forgetting one graph from a graph diffusion
model? That is an experiment the thesis can run.