Extracting Training Data from Diffusion Models (Carlini et al.)

Why

Establishes the threat model behind diffusion unlearning: diffusion models memorise individual
training images and reproduce them at generation time. Without this, “unlearning a training graph”
would be a solution without a problem.

Summary

Bibliographic. Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag,
Florian Tramèr, Borja Balle, Daphne Ippolito, Eric Wallace. Extracting Training Data from Diffusion
Models.
arXiv:2301.13188v1, 30 Jan 2023. 31 pages.

Memorisation definition

  • An example is extractable from a model if an adversary can recover (or a
    near-duplicate) by querying the model.
  • is memorised if it is extractable and the number of near-duplicates of in the
    training set is a small fraction of the dataset. What counts as “small” is left open in the
    paper; it reports results under several thresholds (e.g. distance 0.05 / 0.15).
  • Near-duplicates are found by embedding images with CLIP and comparing cosine similarity, because
    all-pairs pixel comparison over 160 million images is intractable (CLIP gives ~1500Ă— speedup).

Extraction attack

A two-stage generate-and-filter pipeline:

  1. Generate many candidates from known prompts (500 candidates per prompt, using the
    timestep-resampled Stable Diffusion sampler for ~10Ă— speedup).
  2. Filter with a black-box membership-inference heuristic: connect two generations if their
    tile-wise distance is small; if the largest clique has at least 10 of the 500
    generations, predict that the clique is a memorised image. Tile-wise distance (16 tiles of
    128Ă—128) is used instead of full-image because backgrounds create spurious similarity.

Scale of the Stable Diffusion experiment: 350,000 most-duplicated prompts, 175 million generated
images; 94 images were -extracted in one run, and the paper reports over a thousand
extracted training examples overall.

Findings

  • Diffusion models memorise individual training examples and emit them at generation time, from
    photographs of people to trademarked logos.
  • They are much less private than prior generative models such as GANs (the abstract’s
    conclusion).
  • Duplication drives memorisation: examples duplicated many times in the training set are far
    more likely to be extracted. The attack deliberately searches the most-duplicated examples first.
  • Model and training factors matter: the paper trains hundreds of diffusion models to study
    model size, training time, dataset size, augmentation, and deduplication. Memorisation increases
    with capacity/training and with duplication.
  • Deduplication mitigates but is not a complete fix: it reduces extraction, yet non-duplicated
    examples can still be memorised.
  • DP-SGD was not a practical fix in their setup: applying it to CIFAR-10 caused training to
    diverge even at a privacy budget around 50.
  • The paper recommends deduplicating training data, limiting duplication, and developing better
    privacy-preserving training.

Limitations

  • The attack targets image diffusion and uses CLIP-based near-duplicate search. The reported rates
    come from a search biased toward the most-duplicated prompts, so they are not a uniform estimate
    over the training set.
  • The memorisation definition is intentionally narrow (verbatim/near-verbatim reproduction), not
    ”the model contains information about the data”.

Thesis hook

  • This is the empirical justification for the whole thesis: if diffusion models memorise training
    records, a deletion request must actually remove the record’s influence.
  • It supplies the evaluation idea for the graph case: an extraction attack on a graph diffusion
    model would try to reconstruct a training graph (or its subgraph/motif) from samples. The
    ”near-duplicate clique” has a graph analogue: repeatedly sample and test whether the same graph
    (or a graph with high kernel similarity) appears.
  • Duplication → memorisation suggests a confound for the topology experiment: if a graph family is
    more likely to contain repeated motifs, it may look more memorised for reasons unrelated to
    topology. The thesis should control for motif duplication.
  • The failure of DP-SGD in their setup is a warning that a simple DP fix is not available for the
    graph-diffusion stage.

Open questions

  • What is the graph analogue of “verbatim extraction”: exact graph, isomorphic graph, or subgraph?
  • How do we measure near-duplicates of graphs: graph edit distance, WL kernel, MMD, or a learned
    graph embedding?
  • Does graph diffusion memorise high-degree nodes/edges more than low-degree ones? That would link
    memorisation directly to the ER/BA/WS topology question.