Knowledge Unlearning via Gradient Ascent (Jang et al.)
Why
The simplest unlearning baseline, and the one the ER notebook calls NegGrad: take gradient ascent
steps on the data to forget. This paper shows when that is enough and when it breaks.
Summary
Bibliographic. Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen
Logeswaran, Minjoon Seo. Knowledge Unlearning for Mitigating Privacy Risks in Language Models.
Proceedings of the 61st Annual Meeting of the ACL (July 2023), arXiv version in
98 Background Papers/KnowledgeUnlearningGradientAscent.pdf.
Method
Perform gradient ascent on the target token sequences: increase the model’s loss (decrease its
likelihood) on the exact sequence to be forgotten, instead of minimising loss as in pretraining.
No retain-set term is added; the hope is that the update is local enough that general capability
survives.
Setup
- Models: GPT-Neo 125M, 1.3B, 2.7B, pretrained on the Pile. OPT 125M, 1.3B, 2.7B serve as
deduplicated baselines (they were trained on a deduplicated Pile subset), plus DP Decoding as a
second baseline. - Target data: the Training Data Extraction Challenge 2 set — 15,000 examples of 200 tokens
from 16 Pile domains that are easy to extract. By default samples are forgotten at
once, averaged over 5 random samplings. - Metrics:
- EL10 — extraction likelihood: n-gram overlap between the model’s continuation and the
target sequence, averaged over prefixes (n = 10). Lower is better. - MA — memorisation accuracy: fraction of positions where the model’s argmax equals the next
target token. Lower is better. - Forgetting threshold — the mean EL10 and MA on 10,000 held-out Pile validation sequences;
an example counts as forgotten when it falls below both. - Capability — average accuracy on 9 classification benchmarks and average F1 on 4 dialogue
benchmarks.
- EL10 — extraction likelihood: n-gram overlap between the model’s continuation and the
- Optimisation: learning rate 5e-5, no dropout, no weight decay, greedy decoding.
Main results (Table 2, , average of 5 runs)
| Model | EL10 before → after | MA before → after | Classification avg | Dialogue F1 |
|---|---|---|---|---|
| GPT-Neo 125M | 30.9% → 1.0% | 77.4% → 27.4% | 43.4 → 39.9 | 9.4 → 2.6 |
| GPT-Neo 1.3B | 67.6% → 1.9% | 92.2% → 30.4% | 49.8 → 49.7 | 11.5 → 8.5 |
| GPT-Neo 2.7B | 70.4% → 1.6% | 93.4% → 31.0% | 52.3 → 51.9 | 11.5 → 10.8 |
(“before” is the original GPT-Neo; “after” is the + UL+ run, unlearned until the target
sequences meet the forgetting threshold.)
Findings
- Gradient ascent works, and works better on larger models. The 2.7B model keeps essentially
all classification accuracy and most dialogue F1; the 125M model degrades noticeably. - Sequential unlearning beats batch unlearning. Forgetting 128 samples at once ()
damages average performance regardless of model size. Splitting the same 128 samples into four
chunks of 32 and unlearning them sequentially causes almost no degradation. Chunks already
forgotten stay forgotten, and later chunks are forgotten faster. ==🔴important: all samples at once unlearn = bad, in chunks it is good. == - Difficulty depends on the domain. With the 1.3B model, 10 epochs of unlearning on 8 samples:
Enron emails cost only −0.4% average performance, USPTO backgrounds −4.5%. Freelaw is hardest to
forget (final EL10 12.1%, versus 0.0% for most other domains). Structured domains (emails, code)
degrade less than unstructured prose (PubMed Central). - Efficiency. For the 2.7B model the method is about 3,500,000× cheaper than retraining in
FLOPs — floating-point operations, a count of total compute, not wall-clock time. The paper
estimates it as6 × training tokens × parameter count(footnote 5). - DP Decoding reaches zero EL10 but destroys capability (classification N/A, dialogue F1
6.9–7.3), which illustrates the privacy/utility trade-off.
Limitations
- The forgetting threshold depends on the choice of validation set . EL10 and MA are compared
with their means on 10,000 held-out Pile sequences. Change the domain mix, sequence length, sample,
or model, and the threshold moves; the same target can be called forgotten under one and not
under another. The paper’s Table 1 thresholds are EL10 4.99 / 5.68 / 5.53 and MA 29.94 / 33.27 /
34.02 for 125M / 1.3B / 2.7B. Fix before experiments and never tune it to get the desired
verdict. - Prefix length = number of tokens given to the model before asking it to continue. Longer
prefix = more context = easier extraction. The paper varies this as a crude attack-strength knob.
Real extraction attacks can adapt prompts, query many times, sample or beam-search, and filter
outputs; they may be stronger than this fixed-prefix, greedy-decoding check. - Token sequence, not a document. They forget 200-token chunks, not whole documents. Other parts
of the document, long-range context, and structured data (tables, code, graphs) are not covered.
Thesis hook
- The ER notebook’s NegGrad baseline is exactly this method, applied to node labels instead of
tokens. This paper predicts it will be cheap, effective on the forget set, and blunt on retain
performance. - Sequential > batch is a direct design lesson for graph deletion requests: remove one graph at
a time rather than a batch of graphs. - Domain dependence is the closest analogue to the thesis’s topology question: some data is
inherently harder to forget. Replace “domain” with “graph family / degree / clustering” and the
research question is the same. - EL10/MA suggest a graph-side evaluation: likelihood of the forgotten graph under the model versus
held-out graphs, rather than only classification accuracy.
Open questions
- What is the graph analogue of a “forgetting threshold”: NLL of the forgotten graph, edge-level
reconstruction accuracy, or an MMD distance to held-out graphs? - Does sequential unlearning of several graphs also beat batch deletion in a graph diffusion model?
- If domain structure changes forgettability, does graph structure (ER vs BA vs WS) change it in the
same way? This is the thesis’s core hypothesis.