SISS — Data Unlearning in Diffusion Models (Alberti et al.)
Why
A strong current template for removing a specific training datapoint from a diffusion model
without retraining. If the thesis moves from GNN unlearning to graph diffusion (DiGress), this is
the objective to adapt.
Summary
Bibliographic. Silas Alberti, Kenan Hasanaliyev, Manav Shah, Stefano Ermon (Stanford).
Data Unlearning in Diffusion Models. ICLR 2025. arXiv:2503.01034. Code:
https://github.com/claserken/SISS.
Setting
Data unlearning, not concept unlearning: remove specific training datapoints, with no anchor
prompt, class, or target distribution to guide the process. The gold standard is retraining without
those datapoints. Existing concept-unlearning methods need an anchor; general-purpose unlearning
methods were found unstable or ineffective in this setting.
Method
Start from the naive deletion loss on the retained data :
The second term is the NegGrad objective up to a constant. So naive deletion is “retain loss
minus forget loss”, but computing both requires two forward passes.
SISS (Subtracted Importance Sampled Scores) combines them with importance sampling and a
defensive mixture distribution:
Key properties:
- Unbiasedness: for every
in expectation, and the gradient estimators match in expectation (Lemma 1).
This is the paper’s “first method with theoretical guarantees” for data unlearning. - One forward pass per batch instead of two.
- interpolates the behaviour: behaves like naive deletion,
like NegGrad, and is used in practice to keep both benefits. - A superfactor multiplies the forget term by to strengthen unlearning; the
paper studies its effect on stability and interpretability in Appendix A.1.
Experiments
- CelebA-HQ: 30,000 celebrity faces; unconditional DDPM from Ho et al.; 6 celebrity faces
unlearned separately; 60 fine-tuning steps. Metrics: FID, NLL, SSCD similarity. - MNIST T-Shirt: unconditional DDPM on MNIST with 1% Fashion-MNIST T-shirts; a single
datapoint; metrics: Inception Score and exact likelihood. - Stable Diffusion: memorisation prompts.
Results
CelebA-HQ (Table 1):
| Method | FID ↓ | NLL ↑ | SSCD ↓ |
|---|---|---|---|
| Pretrained | 30.3 | 1.257 | 0.87 |
| Naive deletion | 19.6 | 1.240 | 0.87 |
| SISS () | 20.1 | 1.241 | 0.87 |
| SISS () | 25.1 | 1.442 | 0.36 |
| SISS (No IS) | 20.1 | 1.592 | 0.32 |
| EraseDiff | 117.8 | 4.445 | 0.19 |
| SISS () | 327.8 | 6.182 | 0.02 |
| NegGrad | 334.3 | 6.844 | 0.02 |
- SSCD drops by more than 50% while FID stays close to the pretrained model: the model stops
reproducing the celebrity without collapsing. - SISS is Pareto-optimal on the quality/unlearning frontier for CelebA-HQ and MNIST.
- Unlearning 50 celebrity faces sequentially, 60 steps each, ends at FID 20.3 — stable over time.
- On Stable Diffusion, SISS mitigated memorisation on nearly 90% of the tested prompts.
- In contrast, EraseDiff and NegGrad unlearn strongly but destroy quality (FID hundreds), and
naive deletion / leave SSCD unchanged.
Limitations
- Requires access to the pretrained model and the forget set.
- The theoretical guarantee is unbiasedness of the objective, not a differential-privacy bound on
the resulting model. - Evaluated on image diffusion; discrete graph diffusion is not covered.
Thesis hook
- The objective is directly portable in spirit: “retain loss minus forget loss” with importance
sampling. For a graph diffusion model the analogue is an -prediction loss on retained
graphs minus the same loss on the forgotten graph, with balancing them. - The evaluation trio — quality (FID), unlearning strength (SSCD / NLL), likelihood — is a
model for the thesis’s graph metrics (quality: MMD / degree-distribution distance; unlearning:
NLL of the forgotten graph; similarity: graph kernel or edit distance). - The and superfactor are the two knobs the graph version will need.
- SISS and ReTrack are the two diffusion baselines to compare in the later batch.
Open questions
- What is importance sampling over discrete graph diffusion trajectories? DiGress uses categorical
noise; the mixture has to be redefined. - Does the unbiasedness argument survive the graph transformer’s message passing?
- How do we define a “celebrity face” analogue for graphs: one stored graph, one motif, or one
graph family?