Extracting Training Data from Diffusion Models (Carlini et al.)
Why
Establishes the threat model behind diffusion unlearning: diffusion models memorise individual
training images and reproduce them at generation time. Without this, “unlearning a training graph”
would be a solution without a problem.
Summary
Bibliographic. Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag,
Florian Tramèr, Borja Balle, Daphne Ippolito, Eric Wallace. Extracting Training Data from Diffusion
Models. arXiv:2301.13188v1, 30 Jan 2023. 31 pages.
Memorisation definition
- An example is extractable from a model if an adversary can recover (or a
near-duplicate) by querying the model. - is memorised if it is extractable and the number of near-duplicates of in the
training set is a small fraction of the dataset. What counts as “small” is left open in the
paper; it reports results under several thresholds (e.g. distance 0.05 / 0.15). - Near-duplicates are found by embedding images with CLIP and comparing cosine similarity, because
all-pairs pixel comparison over 160 million images is intractable (CLIP gives ~1500Ă— speedup).
Extraction attack
A two-stage generate-and-filter pipeline:
- Generate many candidates from known prompts (500 candidates per prompt, using the
timestep-resampled Stable Diffusion sampler for ~10Ă— speedup). - Filter with a black-box membership-inference heuristic: connect two generations if their
tile-wise distance is small; if the largest clique has at least 10 of the 500
generations, predict that the clique is a memorised image. Tile-wise distance (16 tiles of
128Ă—128) is used instead of full-image because backgrounds create spurious similarity.
Scale of the Stable Diffusion experiment: 350,000 most-duplicated prompts, 175 million generated
images; 94 images were -extracted in one run, and the paper reports over a thousand
extracted training examples overall.
Findings
- Diffusion models memorise individual training examples and emit them at generation time, from
photographs of people to trademarked logos. - They are much less private than prior generative models such as GANs (the abstract’s
conclusion). - Duplication drives memorisation: examples duplicated many times in the training set are far
more likely to be extracted. The attack deliberately searches the most-duplicated examples first. - Model and training factors matter: the paper trains hundreds of diffusion models to study
model size, training time, dataset size, augmentation, and deduplication. Memorisation increases
with capacity/training and with duplication. - Deduplication mitigates but is not a complete fix: it reduces extraction, yet non-duplicated
examples can still be memorised. - DP-SGD was not a practical fix in their setup: applying it to CIFAR-10 caused training to
diverge even at a privacy budget around 50. - The paper recommends deduplicating training data, limiting duplication, and developing better
privacy-preserving training.
Limitations
- The attack targets image diffusion and uses CLIP-based near-duplicate search. The reported rates
come from a search biased toward the most-duplicated prompts, so they are not a uniform estimate
over the training set. - The memorisation definition is intentionally narrow (verbatim/near-verbatim reproduction), not
”the model contains information about the data”.
Thesis hook
- This is the empirical justification for the whole thesis: if diffusion models memorise training
records, a deletion request must actually remove the record’s influence. - It supplies the evaluation idea for the graph case: an extraction attack on a graph diffusion
model would try to reconstruct a training graph (or its subgraph/motif) from samples. The
”near-duplicate clique” has a graph analogue: repeatedly sample and test whether the same graph
(or a graph with high kernel similarity) appears. - Duplication → memorisation suggests a confound for the topology experiment: if a graph family is
more likely to contain repeated motifs, it may look more memorised for reasons unrelated to
topology. The thesis should control for motif duplication. - The failure of DP-SGD in their setup is a warning that a simple DP fix is not available for the
graph-diffusion stage.
Open questions
- What is the graph analogue of “verbatim extraction”: exact graph, isomorphic graph, or subgraph?
- How do we measure near-duplicates of graphs: graph edit distance, WL kernel, MMD, or a learned
graph embedding? - Does graph diffusion memorise high-degree nodes/edges more than low-degree ones? That would link
memorisation directly to the ER/BA/WS topology question.