IterNorm-pytorch

Pytorch reimplementation of the IterNorm methods, which is described in the following paper:

Iterative Normalization: Beyond Standardization towards Efficient Whitening

Lei Huang, Yi Zhou, Fan Zhu, Li Liu, Ling Shao

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (accepted). arXiv:1904.03441

This project also provide the pytorch implementation of Decorrelated Batch Normalization (CVPR 2018, arXiv:1804.08450), more details please refer to the Torch project.

Requirements and Dependency

• Install PyTorch with CUDA (for GPU). (Experiments are validated on python 3.6.8 and pytorch-nightly 1.0.0)
• (For visualization if needed), install the dependency visdom by:

pip install visdom

Experiments

1. VGG-network on Cifar-10 datasets:

run the scripts in the ./cifar10/experiments/vgg. Note that the dataset root dir should be altered by setting the para '--dataset-root', and the dataset style is described as:

-<dataset-root>
|-cifar10-batches-py
||-data_batch_1
||-data_batch_2
||-data_batch_3
||-data_batch_4
||-data_batch_5
||-test_batch

If the dataset is not exist, the script will download it, under the conditioning that the dataset-root dir is existed

2. Wide-Residual-Network on Cifar-10 datasets:

run the scripts in the ./cifar10/experiments/wrn.

3. ImageNet experiments.

run the scripts in the ./ImageNet/experiment. Note that resnet18 experimetns are run on one GPU, and resnet-50/101 are run on 4 GPU in the scripts.

Note that the dataset root dir should be altered by setting the para '--dataset-root'. and the dataset style is described as:

-<dataset-root>
|-train
||-class1
||-...
||-class1000  
|-var
||-class1
||-...
||-class1000  

Using IterNorm in other projects/tasks

(1) copy ./extension/normalization/iterative_normalization.py to the respective dir.

(2) import the IterNorm class in iterative_normalization.py

(3) generally speaking, replace the BatchNorm layer by IterNorm, or add it in any place if you want to the feature/channel decorrelated. Considering the efficiency (Note that BatchNorm is intergrated in cudnn while IterNorm is based on the pytorch script without optimization), we recommend 1) replace the first BatchNorm; 2) insert extra IterNorm before the first skip connection in resnet; 3) inserted before the final linear classfier as described in the paper.

(4) Some tips related to the hyperparamters (Group size G and Iterative Number T). We recommend G=64 (i.e., the channel number in per group is 64) and T=5 by default. If you run on large batch size (e.g.>1024), you can either increase G or T. For fine tunning, fix G=64 or G=32, and search T={3,4,5,6,7,8} may help.