README.md

Network compression can reduce the memory footprint of a neural network, increase its inference speed and save energy. Distiller provides a PyTorch environment for prototyping and analyzing compression algorithms, such as sparsity-inducing methods and low-precision arithmetic.

Changed the YAML API to express complex dependencies when pruning channels and filters.

Fixed a bunch of bugs

Image classifier compression sample:

Added a new command-line argument to report the top N best accuracy scores, instead of just the highest score.

Added an option to load a model in serialized mode.

We've fixed a couple of Early Exit bugs, and improved the documentation

We presented Distiller at AIDC 2018 Beijing and @haim-barad presented his Early Exit research implemented using Distiller.

We've looked up our star-gazers (that might be you ;-) and where they are located:The map was generated by this utility.

What's New in October?

We've added two new Jupyter notebooks:

The first notebook contrasts what sparse and dense versions of ResNet50 "look at".

The second notebook shows a simple application of Truncated SVD to the linear layer in ResNet50.

We've added collection of activation statistics!

Activation statistics can be leveraged to make pruning and quantization decisions, and so
we added support to collect these data.
Two types of activation statistics are supported: summary statistics, and detailed records
per activation.
Currently we support the following summaries:

Average activation sparsity, per layer

Average L1-norm for each activation channel, per layer

Average sparsity for each activation channel, per layer

For the detailed records we collect some statistics per activation and store it in a record.
Using this collection method generates more detailed data, but consumes more time, so
Beware.

You can collect activation data for the different training phases: training/validation/test.

You can access the data directly from each module that you chose to collect stats for.

The compression schedule is expressed in a YAML file so that a single file captures the details of experiments. This dependency injection design decouples the Distiller scheduler and library from future extensions of algorithms.

Element-wise and filter-wise pruning sensitivity analysis (using L1-norm thresholding). Examine the data from some of the networks we analyzed, using this notebook.

Regularization

L1-norm element-wise regularization

Group Lasso an group variance regularization

Quantization

Automatic mechanism to transform existing models to quantized versions, with customizable bit-width configuration for different layers. No need to re-write the model for different quantization methods.

Installation

Distiller has only been tested on Ubuntu 16.04 LTS, and with Python 3.5.

If you are not using a GPU, you might need to make small adjustments to the code.

Clone Distiller

Clone the Distiller code repository from github:

$ git clone https://github.com/NervanaSystems/distiller.git

The rest of the documentation that follows, assumes that you have cloned your repository to a directory called distiller.

Create a Python virtual environment

We recommend using a Python virtual environment, but that of course, is up to you.
There's nothing special about using Distiller in a virtual environment, but we provide some instructions, for completeness.
Before creating the virtual environment, make sure you are located in directory distiller. After creating the environment, you should see a directory called distiller/env.

Using virtualenv

If you don't have virtualenv installed, you can find the installation instructions here.

To create the environment, execute:

$ python3 -m virtualenv env

This creates a subdirectory named env where the python virtual environment is stored, and configures the current shell to use it as the default python environment.

Using venv

If you prefer to use venv, then begin by installing it:

$ sudo apt-get install python3-venv

Then create the environment:

$ python3 -m venv env

As with virtualenv, this creates a directory called distiller/env.

Activate the environment

The environment activation and deactivation commands for venv and virtualenv are the same.!NOTE: Make sure to activate the environment, before proceeding with the installation of the dependency packages:

$ source env/bin/activate

Install dependencies

Finally, install Distiller's dependency packages using pip3:

$ pip3 install -r requirements.txt

PyTorch is included in the requirements.txt file, and will currently download PyTorch version 0.4.0 for CUDA 8.0. This is the setup we've used for testing Distiller.

Getting Started

You can jump head-first into some limited examples of network compression, to get a feeling for the library without too much investment on your part.

Distiller comes with a sample application for compressing image classification DNNs, compress_classifier.py located at distiller/examples/classifier_compression.

We'll show you how to use it for some simple use-cases, and will point you to some ready-to-go Jupyter notebooks.

Example invocations of the sample application

Training-only

The following will invoke training-only (no compression) of a network named 'simplenet' on the CIFAR10 dataset. This is roughly based on TorchVision's sample Imagenet training application, so it should look familiar if you've used that application. In this example we don't invoke any compression mechanisms: we just train because for fine-tuning after pruning, training is an essential part.

Note that the first time you execute this command, the CIFAR10 code will be downloaded to your machine, which may take a bit of time - please let the download process proceed to completion.

The path to the CIFAR10 dataset is arbitrary, but in our examples we place the datasets in the same directory level as distiller (i.e. ../../../data.cifar10).

You can use a TensorBoard backend to view the training progress (in the diagram below we show a couple of training sessions with different LR values). For compression sessions, we've added tracing of activation and parameter sparsity levels, and regularization loss.

Getting parameter statistics of a sparsified model

We've included in the git repository a few checkpoints of a ResNet20 model that we've trained with 32-bit floats. Let's load the checkpoint of a model that we've trained with channel-wise Group Lasso regularization.
With the following command-line arguments, the sample application loads the model (--resume) and prints statistics about the model weights (--summary=sparsity). This is useful if you want to load a previously pruned model, to examine the weights sparsity statistics, for example. Note that when you resume a stored checkpoint, you still need to tell the application which network architecture the checkpoint uses (-a=resnet20_cifar):

You should see a text table detailing the various sparsities of the parameter tensors. The first column is the parameter name, followed by its shape, the number of non-zero elements (NNZ) in the dense model, and in the sparse model. The next set of columns show the column-wise, row-wise, channel-wise, kernel-wise, filter-wise and element-wise sparsities.
Wrapping it up are the standard-deviation, mean, and mean of absolute values of the elements.

In the Compression Insights notebook we use matplotlib to plot a bar chart of this summary, that indeed show non-impressive footprint compression.

Although the memory footprint compression is very low, this model actually saves 26.6% of the MACs compute.

Post-training quantization

This example performs 8-bit quantization of ResNet20 for CIFAR10. We've included in the git repository the checkpoint of a ResNet20 model that we've trained with 32-bit floats, so we'll take this model and quantize it:

The command-line above will save a checkpoint named quantized_checkpoint.pth.tar containing the quantized model parameters. See more examples here.

Explore the sample Jupyter notebooks

The set of notebooks that come with Distiller is described here, which also explains the steps to install the Jupyter notebook server.
After installing and running the server, take a look at the notebook covering pruning sensitivity analysis.

Sensitivity analysis is a long process and this notebook loads CSV files that are the output of several sessions of sensitivity analysis.

Set up the classification datasets

The sample application for compressing image classification DNNs, compress_classifier.py located at distiller/examples/classifier_compression, uses both CIFAR10 and ImageNet image datasets.

The compress_classifier.py application will download the CIFAR10 automatically the first time you try to use it (thanks to TorchVision). The example invocations used throughout Distiller's documentation assume that you have downloaded the images to directory distiller/../data.cifar10, but you can place the images anywhere you want (you tell compress_classifier.py where the dataset is located - or where you want the application to download the dataset to - using a command-line parameter).

ImageNet needs to be downloaded manually, due to copyright issues. Facebook has created a set of scripts to help download and extract the dataset.

Again, the Distiller documentation assumes the following directory structure for the datasets, but this is just a suggestion:

Running the tests

We are currently light-weight on test and this is an area where contributions will be much appreciated.
There are two types of tests: system tests and unit-tests. To invoke the unit tests:

$ cd distiller/tests
$ pytest

We use CIFAR10 for the system tests, because its size makes for quicker tests. To invoke the system tests, you need to provide a path to the CIFAR10 dataset which you've already downloaded. Alternatively, you may invoke full_flow_tests.py without specifying the location of the CIFAR10 dataset and let the test download the dataset (for the first invocation only). Note that --cifar1o-path defaults to the current directory.
The system tests are not short, and are even longer if the test needs to download the dataset.

Acknowledgments

Any published work is built on top of the work of many other people, and the credit belongs to too many people to list here.

The Python and PyTorch developer communities have shared many invaluable insights, examples and ideas on the Web.

The authors of the research papers implemented in the Distiller model-zoo have shared their research ideas, theoretical background and results.

Disclaimer

Distiller is released as a reference code for research purposes. It is not an official Intel product, and the level of quality and support may not be as expected from an official product. Additional algorithms and features are planned to be added to the library. Feedback and contributions from the open source and research communities are more than welcome.