Dataset Distillation

Tongzhou Wang12    Jun-Yan Zhu2    Antonio Torralba2    Alexei A. Efros3   

1Facebook AI Research    2MIT CSAIL    3UC Berkeley

Paper | PyTorch code

Check out our latest CVPR 2022 work: Project | Paper | PyTorch code


Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to original performance with only a few steps of gradient descent, given a fixed network initialization. We evaluate our method in various initialization settings and with different learning objectives. Experiments on multiple datasets show the advantage of our approach compared to alternative methods.

paper thumbnail


arxiv 1811.10959, 2018.


Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A. Efros. "Dataset Distillation", arXiv preprint, 2018. Bibtex

Code: GitHub

Experiment Results

Standard devaiations mentioned below are calculated on 200 held-out models.

Train networks with a fixed known initialization


10 images train test accuracy from 12.9% to 93.8%


100 images train test accuracy from 8.8% to 54.0%


Train networks with unknown random initializations


100 images train test accuracy to 79.5% ± 8.1%


100 images train test accuracy to 36.8% ± 1.2%


Adapt pre-trained networks with unknown weights to a new dataset


100 images train test accuracy
from 67.5% ± 3.9% to 92.7% ± 1.4%


100 images train test accuracy
from 51.6% ± 2.8% to 85.2% ± 4.7%


Attack well-trained classifiers with unknown weights within 1 gradient step


100 images train classifiers with 98.6% ± 0.5% test accuracy
to predict 71.4% ± 29.6% label 0 test images as label 1

CIFAR10: planecar

100 images train classifiers with 78.2% ± 1.1% test accuracy
to predict 45.9% ± 18.1% label plane test images as label car


See our paper for more experiments, including adapting an AlexNet (pre-trained on ImageNet) to PASCAL-POC and CUB-200 with only one image per class.

Related Work


This work was supported in part by NSF 1524817 on Advancing Visual Recognition with Feature Visualizations, NSF IIS-1633310, and Berkeley Deep Drive.