Pytorch Implementation of "Desigining Network Design Spaces", Radosavovic et al. CVPR 2020.

Overview

RegNet

Designing Network Design Spaces

Pytorch Implementation of "Desigining Network Design Spaces", Radosavovic et al. CVPR 2020.

Paper | Official Implementation

RegNet offer a very nice design space for neural network architectures. RegNet design space consists of networks with simple structure which authors call "Regular" Networks (RegNet). Models in RegNet design space have higher concentration of models that perform well and generalise well. RegNet models are very efficient and run upto 5 times faster than EfficientNet models on GPUs.

Also RegNet models have been used as a backbone in Tesla FSD Stack.

Overview Of AnyNet

  • Main goal of the paper is to help in better understanding of network design and discover principles that generalize across settings.
  • Explore structure aspeck of network design and arrive at low dimensional design space consisting of simple regualar networks
  • Network width and depth can be explained by a quantized linear function.

AnyNet Design Space

The basic structure of models in AnyNet design space consists of a simple Stem which is then followed by the network body that does majority of the computation and a final network head that predicts the class scores. The stem and head networks are kept as simple as possible. The network body consists of 4 stages that operate at progressively lower resolutions.

AnyNet

Structure of network body is determined by block width w, network depth d_i, bottleneck ratio b_i and group widths g. Degrees of freedom at stage 'i' are number of blocks d in each stage, block width w and other block parameters such as stride, padding and so on.

Other models are obtained by refining the design space by adding more constraints on the above parameters. Design space is refined keeping the following things in mind :

  • Simplify structure of design space.
  • Improve the interpretability of design space.
  • Maintain Design space complexity.
  • Maintain model diversity in design space.

AnyNetX

XBlock

  • Uses XBlocks within each block of the network
  • Degrees of freedom in AnyNetX is 16
  • Each network has 4 stages
  • Each stage has 4 parameters (network depth di, block width wi, bottleneck ratio bi, group width gi)
  • bi ∈ {1,2,4}
  • gi ∈ {1,2,3,...,32}
  • wi <= 1024
  • di <= 16

AnyNetX(A)

AnyNetX(A) is same as the above AnyNetX

AnyNetX(B)

In this design space,

  • bottleneck ratio bi is fixed for all stages.
  • performance of models in AnyNetX(B) space is almost equal to AnyNetX(A) in average and best case senarios
  • bi <= 2 seemes to work best.

AnyNetX(C)

In this design space,

  • Shared group width gi for all stages.
  • AnyNetX(C) has 6 fewer degrees of freedom compared to AnyNetX(A)
  • gi > 1 seems to work best

AnyNetX(D)

In AnyNetX(D) design space, authors observed that good networks have increasing stage widths w(i+1) > wi

AnyNetX(E)

In AnyNetX(E) design space, it was observed that as stage widths wi increases, depth di likewise tend to increase except for the last stage.

RegNet

Please refer to Section 3.3 in paper.

Training

Import any of the following variants of RegNet using

from regnet import regnetx_002 as RegNet002
from regnet import Xblock, Yblock # required if you want to use YBlock instead of Xblock. Refer to paper for more details on YBlock

RegNet variants available are:

  • regnetx_002
  • regnetx_004
  • regnetx_006
  • regnetx_008
  • regnetx_016
  • regnetx_032
  • regnetx_040
  • regnetx_064
  • regnetx_080
  • regnetx_120
  • regnetx_160
  • regnetx_320

Import TrainingConfig and Trainer Classes from regnet and use them to train the model as follows

from regnet import TrainingConfig, Trainer

model = RegNet002(block=Xblock, num_classes=10)

training_config = TrainingConfig(max_epochs=10, batch_size=128, learning_rate=3e-4, weight_decay=5e-4, ckpt_path="./regnet.pt")
trainer = Trainer(model = model, train_dataset=train_dataset, test_dataset=test_dataset, config=training_config)
trainer.train()

Note : you need not use TrainingConfig and Trainer classes if you want to write your own training loops. Just importing the respective models would suffice.

TODO

  • Test if model trains when using YBlocks
  • Implement model checkpointing for every 'x' epochs

References

[1] https://github.com/signatrix/regnet

[2] https://github.com/d-li14/regnet.pytorch

@InProceedings{Radosavovic2020,
  title = {Designing Network Design Spaces},
  author = {Ilija Radosavovic and Raj Prateek Kosaraju and Ross Girshick and Kaiming He and Piotr Doll{\'a}r},
  booktitle = {CVPR},
  year = {2020}
}

LICENSE

MIT

Owner
Vishal R
Computer Science Student at PES University.
Vishal R
Recommendationsystem - Movie-recommendation - matrixfactorization colloborative filtering recommendation system user

recommendationsystem matrixfactorization colloborative filtering recommendation

kunal jagdish madavi 1 Jan 01, 2022
This is a library for training and applying sparse fine-tunings with torch and transformers.

This is a library for training and applying sparse fine-tunings with torch and transformers. Please refer to our paper Composable Sparse Fine-Tuning f

Cambridge Language Technology Lab 37 Dec 30, 2022
A PyTorch Implementation of the Luna: Linear Unified Nested Attention

Unofficial PyTorch implementation of Luna: Linear Unified Nested Attention The quadratic computational and memory complexities of the Transformer’s at

Soohwan Kim 32 Nov 07, 2022
StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion

StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion Yinghao Aaron Li, Ali Zare, Nima Mesgarani We pres

Aaron (Yinghao) Li 282 Jan 01, 2023
GPU-accelerated PyTorch implementation of Zero-shot User Intent Detection via Capsule Neural Networks

GPU-accelerated PyTorch implementation of Zero-shot User Intent Detection via Capsule Neural Networks This repository implements a capsule model Inten

Joel Huang 15 Dec 24, 2022
HiPAL: A Deep Framework for Physician Burnout Prediction Using Activity Logs in Electronic Health Records

HiPAL Code for KDD'22 Applied Data Science Track submission -- HiPAL: A Deep Framework for Physician Burnout Prediction Using Activity Logs in Electro

Hanyang Liu 4 Aug 08, 2022
Groceries ARL: Association Rules (Birliktelik Kuralı)

Groceries_ARL Association Rules (Birliktelik Kuralı) Birliktelik kuralları, mark

Şebnem 5 Feb 08, 2022
Mall-Customers-Segmentation - Customer Segmentation Using K-Means Clustering

Overview Customer Segmentation is one the most important applications of unsupervised learning. Using clustering techniques, companies can identify th

NelakurthiSudheer 2 Jan 03, 2022
A Kitti Road Segmentation model implemented in tensorflow.

KittiSeg KittiSeg performs segmentation of roads by utilizing an FCN based model. The model achieved first place on the Kitti Road Detection Benchmark

Marvin Teichmann 890 Jan 04, 2023
Explainer for black box models that predict molecule properties

Explaining why that molecule exmol is a package to explain black-box predictions of molecules. The package uses model agnostic explanations to help us

White Laboratory 172 Dec 19, 2022
《Where am I looking at? Joint Location and Orientation Estimation by Cross-View Matching》(CVPR 2020)

This contains the codes for cross-view geo-localization method described in: Where am I looking at? Joint Location and Orientation Estimation by Cross-View Matching, CVPR2020.

41 Oct 27, 2022
FluidNet re-written with ATen tensor lib

fluidnet_cxx: Accelerating Fluid Simulation with Convolutional Neural Networks. A PyTorch/ATen Implementation. This repository is based on the paper,

JoliBrain 50 Jun 07, 2022
The official code of Anisotropic Stroke Control for Multiple Artists Style Transfer

ASMA-GAN Anisotropic Stroke Control for Multiple Artists Style Transfer Proceedings of the 28th ACM International Conference on Multimedia The officia

Six_God 146 Nov 21, 2022
[CVPR'21] FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space

FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space by Quande Liu, Cheng Chen, Ji

Quande Liu 178 Jan 06, 2023
Companion code for the paper "An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their Asymptotic Overconfidence" (NeurIPS 2021)

ReLU-GP Residual (RGPR) This repository contains code for reproducing the following NeurIPS 2021 paper: @inproceedings{kristiadi2021infinite, title=

Agustinus Kristiadi 4 Dec 26, 2021
A Neural Net Training Interface on TensorFlow, with focus on speed + flexibility

Tensorpack is a neural network training interface based on TensorFlow. Features: It's Yet Another TF high-level API, with speed, and flexibility built

Tensorpack 6.2k Jan 09, 2023
This Repostory contains the pretrained DTLN-aec model for real-time acoustic echo cancellation.

This Repostory contains the pretrained DTLN-aec model for real-time acoustic echo cancellation.

Nils L. Westhausen 182 Jan 07, 2023
Official Keras Implementation for UNet++ in IEEE Transactions on Medical Imaging and DLMIA 2018

UNet++: A Nested U-Net Architecture for Medical Image Segmentation UNet++ is a new general purpose image segmentation architecture for more accurate i

Zongwei Zhou 1.8k Jan 07, 2023
A Python implementation of active inference for Markov Decision Processes

A Python package for simulating Active Inference agents in Markov Decision Process environments. Please see our companion preprint on arxiv for an ove

235 Dec 21, 2022
Fader Networks: Manipulating Images by Sliding Attributes - NIPS 2017

FaderNetworks PyTorch implementation of Fader Networks (NIPS 2017). Fader Networks can generate different realistic versions of images by modifying at

Facebook Research 753 Dec 23, 2022