Network Pruning that Matters: A Case Study on Retraining Variants

This repository contains the implementation of the paper Network Pruning that Matters: A Case Study on Retraining Variants.

Duong H. Le, Binh-Son Hua (ICLR 2021)

In this work, we study the behavior of pruned networks under different retraining settings. By leveraging the right learning rate schedule in retraining, we demonstrate a counter-intuitive phenomenon in that randomly pruned networks could even achieve better performance than methodically pruned networks (fine-tuned with the conventional approach) in many scenariors. Our results emphasize the cruciality of the learning rate schedule in pruned network retraining – a detail often overlooked by practioners during the implementation of network pruning.

If you find the paper/code helpful, please cite our paper:

@inproceedings{
le2021network,
title={Network Pruning That Matters:  A Case Study on Retraining Variants},
author={Duong Hoang Le and Binh-Son Hua},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=Cb54AMqHQFP}
}

How to Run

To run the code:

Copy the Imagenet/CIFAR-10 dataset to ./data folder
Run init.sh
Download checkpoints here then uncompress it here
Run the desired script in each subfolder.

Acknowledgement

Our implementation is based on the official code of HRank, Taylor Pruning, Soft Filter Pruning, Rethinking.

Network Pruning That Matters: A Case Study on Retraining Variants (ICLR 2021)

Related tags

Overview

Network Pruning that Matters: A Case Study on Retraining Variants

How to Run

Acknowledgement

Owner

Duong H. Le

Official implementation of "UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer"

Github Traffic Insights as Prometheus metrics.

Official Pytorch Implementation of: "ImageNet-21K Pretraining for the Masses"(2021) paper

YOLTv5 rapidly detects objects in arbitrarily large aerial or satellite images that far exceed the ~600×600 pixel size typically ingested by deep learning object detection frameworks

The Agriculture Domain of ERPNext comes with features to record crops and land

Implementation of the "PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences" paper.

CNN Based Meta-Learning for Noisy Image Classification and Template Matching

This is the replication package for paper submission: Towards Training Reproducible Deep Learning Models.

GLANet - The code for Global and Local Alignment Networks for Unpaired Image-to-Image Translation arxiv

Isaac Gym Reinforcement Learning Environments

Official code for our EMNLP2021 Outstanding Paper MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks

A complete end-to-end demonstration in which we collect training data in Unity and use that data to train a deep neural network to predict the pose of a cube. This model is then deployed in a simulated robotic pick-and-place task.

Implementation of Gans

September-Assistant - Open-source Windows Voice Assistant

Official Matlab Implementation for "Tiny Obstacle Discovery by Occlusion-aware Multilayer Regression", TIP 2020

Circuit Training: An open-source framework for generating chip floor plans with distributed deep reinforcement learning

RL-GAN: Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation

Some methods for comparing network representations in deep learning and neuroscience.

Multivariate Time Series Forecasting with efficient Transformers. Code for the paper "Long-Range Transformers for Dynamic Spatiotemporal Forecasting."

Einshape: DSL-based reshaping library for JAX and other frameworks.