Pytorch Implementation of "Desigining Network Design Spaces", Radosavovic et al. CVPR 2020.

Overview

RegNet

Designing Network Design Spaces

Pytorch Implementation of "Desigining Network Design Spaces", Radosavovic et al. CVPR 2020.

Paper | Official Implementation

RegNet offer a very nice design space for neural network architectures. RegNet design space consists of networks with simple structure which authors call "Regular" Networks (RegNet). Models in RegNet design space have higher concentration of models that perform well and generalise well. RegNet models are very efficient and run upto 5 times faster than EfficientNet models on GPUs.

Also RegNet models have been used as a backbone in Tesla FSD Stack.

Overview Of AnyNet

  • Main goal of the paper is to help in better understanding of network design and discover principles that generalize across settings.
  • Explore structure aspeck of network design and arrive at low dimensional design space consisting of simple regualar networks
  • Network width and depth can be explained by a quantized linear function.

AnyNet Design Space

The basic structure of models in AnyNet design space consists of a simple Stem which is then followed by the network body that does majority of the computation and a final network head that predicts the class scores. The stem and head networks are kept as simple as possible. The network body consists of 4 stages that operate at progressively lower resolutions.

AnyNet

Structure of network body is determined by block width w, network depth d_i, bottleneck ratio b_i and group widths g. Degrees of freedom at stage 'i' are number of blocks d in each stage, block width w and other block parameters such as stride, padding and so on.

Other models are obtained by refining the design space by adding more constraints on the above parameters. Design space is refined keeping the following things in mind :

  • Simplify structure of design space.
  • Improve the interpretability of design space.
  • Maintain Design space complexity.
  • Maintain model diversity in design space.

AnyNetX

XBlock

  • Uses XBlocks within each block of the network
  • Degrees of freedom in AnyNetX is 16
  • Each network has 4 stages
  • Each stage has 4 parameters (network depth di, block width wi, bottleneck ratio bi, group width gi)
  • bi ∈ {1,2,4}
  • gi ∈ {1,2,3,...,32}
  • wi <= 1024
  • di <= 16

AnyNetX(A)

AnyNetX(A) is same as the above AnyNetX

AnyNetX(B)

In this design space,

  • bottleneck ratio bi is fixed for all stages.
  • performance of models in AnyNetX(B) space is almost equal to AnyNetX(A) in average and best case senarios
  • bi <= 2 seemes to work best.

AnyNetX(C)

In this design space,

  • Shared group width gi for all stages.
  • AnyNetX(C) has 6 fewer degrees of freedom compared to AnyNetX(A)
  • gi > 1 seems to work best

AnyNetX(D)

In AnyNetX(D) design space, authors observed that good networks have increasing stage widths w(i+1) > wi

AnyNetX(E)

In AnyNetX(E) design space, it was observed that as stage widths wi increases, depth di likewise tend to increase except for the last stage.

RegNet

Please refer to Section 3.3 in paper.

Training

Import any of the following variants of RegNet using

from regnet import regnetx_002 as RegNet002
from regnet import Xblock, Yblock # required if you want to use YBlock instead of Xblock. Refer to paper for more details on YBlock

RegNet variants available are:

  • regnetx_002
  • regnetx_004
  • regnetx_006
  • regnetx_008
  • regnetx_016
  • regnetx_032
  • regnetx_040
  • regnetx_064
  • regnetx_080
  • regnetx_120
  • regnetx_160
  • regnetx_320

Import TrainingConfig and Trainer Classes from regnet and use them to train the model as follows

from regnet import TrainingConfig, Trainer

model = RegNet002(block=Xblock, num_classes=10)

training_config = TrainingConfig(max_epochs=10, batch_size=128, learning_rate=3e-4, weight_decay=5e-4, ckpt_path="./regnet.pt")
trainer = Trainer(model = model, train_dataset=train_dataset, test_dataset=test_dataset, config=training_config)
trainer.train()

Note : you need not use TrainingConfig and Trainer classes if you want to write your own training loops. Just importing the respective models would suffice.

TODO

  • Test if model trains when using YBlocks
  • Implement model checkpointing for every 'x' epochs

References

[1] https://github.com/signatrix/regnet

[2] https://github.com/d-li14/regnet.pytorch

@InProceedings{Radosavovic2020,
  title = {Designing Network Design Spaces},
  author = {Ilija Radosavovic and Raj Prateek Kosaraju and Ross Girshick and Kaiming He and Piotr Doll{\'a}r},
  booktitle = {CVPR},
  year = {2020}
}

LICENSE

MIT

Owner
Vishal R
Computer Science Student at PES University.
Vishal R
An official implementation of "Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation" (ICCV 2021) in PyTorch.

Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation This is an official implementation of the paper "Exploiting a Joint

CV Lab @ Yonsei University 35 Oct 26, 2022
Gym environment for FLIPIT: The Game of "Stealthy Takeover"

gym-flipit Gym environment for FLIPIT: The Game of "Stealthy Takeover" invented by Marten van Dijk, Ari Juels, Alina Oprea, and Ronald L. Rivest. Desi

Lisa Oakley 2 Dec 15, 2021
This repo contains the source code and a benchmark for predicting user's utilities with Machine Learning techniques for Computational Persuasion

Machine Learning for Argument-Based Computational Persuasion This repo contains the source code and a benchmark for predicting user's utilities with M

Ivan Donadello 4 Nov 07, 2022
NaijaSenti is an open-source sentiment and emotion corpora for four major Nigerian languages

NaijaSenti is an open-source sentiment and emotion corpora for four major Nigerian languages. This project was supported by lacuna-fund initiatives. Jump straight to one of the sections below, or jus

Hausa Natural Language Processing 14 Dec 20, 2022
A medical imaging framework for Pytorch

Welcome to MedicalTorch MedicalTorch is an open-source framework for PyTorch, implementing an extensive set of loaders, pre-processors and datasets fo

Christian S. Perone 799 Jan 03, 2023
HPRNet: Hierarchical Point Regression for Whole-Body Human Pose Estimation

HPRNet: Hierarchical Point Regression for Whole-Body Human Pose Estimation Official PyTroch implementation of HPRNet. HPRNet: Hierarchical Point Regre

Nermin Samet 53 Dec 04, 2022
A repository for storing njxzc final exam review material

文档地址,请戳我 👈 👈 👈 ☀️ 1.Reason 大三上期末复习软件工程的时候,发现其他高校在GitHub上开源了他们学校的期末试题,我很受触动。期末

GuJiakai 2 Jan 18, 2022
Coded illumination for improved lensless imaging

CodedCam Coded Illumination for Improved Lensless Imaging Paper | Supplementary results | Data and Code are available. Coded illumination for improved

Computational Sensing and Information Processing Lab 1 Nov 29, 2021
Some methods for comparing network representations in deep learning and neuroscience.

Generalized Shape Metrics on Neural Representations In neuroscience and in deep learning, quantifying the (dis)similarity of neural representations ac

Alex Williams 45 Dec 27, 2022
PyTorch implementation of Self-supervised Contrastive Regularization for DG (SelfReg)

SelfReg PyTorch official implementation of Self-supervised Contrastive Regularization for Domain Generalization (SelfReg, https://arxiv.org/abs/2104.0

64 Dec 16, 2022
PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

PyTorch-LIT PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices. With

Amin Rezaei 157 Dec 11, 2022
Data and code from COVID-19 machine learning paper

Machine learning approaches for localized lockdown, subnotification analysis and cases forecasting in São Paulo state counties during COVID-19 pandemi

Sara Malvar 4 Dec 22, 2022
Code for 'Blockwise Sequential Model Learning for Partially Observable Reinforcement Learning' (AAAI 2022)

Blockwise Sequential Model Learning Code for 'Blockwise Sequential Model Learning for Partially Observable Reinforcement Learning' (AAAI 2022) For ins

2 Jun 17, 2022
Tooling for converting STAC metadata to ODC data model

手语识别 0、使用到的模型 (1). openpose,作者:CMU-Perceptual-Computing-Lab https://github.com/CMU-Perceptual-Computing-Lab/openpose (2). 图像分类classification,作者:Bubbl

Open Data Cube 65 Dec 20, 2022
An open source app to help calm you down when needed.

By: Seanpm2001, Et; Al. Top README.md Read this article in a different language Sorted by: A-Z Sorting options unavailable ( af Afrikaans Afrikaans |

Sean P. Myrick V19.1.7.2 2 Oct 24, 2022
Rotated Box Is Back : Accurate Box Proposal Network for Scene Text Detection

Rotated Box Is Back : Accurate Box Proposal Network for Scene Text Detection This material is supplementray code for paper accepted in ICDAR 2021 We h

NCSOFT 30 Dec 21, 2022
Efficient Speech Processing Tookit for Automatic Speaker Recognition

Sugar Efficient Speech Processing Tookit for Automatic Speaker Recognition | HuggingFace | What's New EfficientTDNN: Efficient Architecture Search for

WangRui 14 Sep 14, 2022
Project dự đoán giá cổ phiếu bằng thuật toán LSTM gồm: code train và code demo

Web predicts stock prices using Long - Short Term Memory algorithm Give me some start please!!! User interface image: Choose: DayBegin, DayEnd, Stock

Vo Thuong Truong Nhon 8 Nov 11, 2022
This is the source code for: Context-aware Entity Typing in Knowledge Graphs.

This is the source code for: Context-aware Entity Typing in Knowledge Graphs.

9 Sep 01, 2022
EmoTag helps you train emotion detection model for Chinese audios

emoTag emoTag helps you train emotion detection model for Chinese audios. Environment pip install -r requirement.txt Data We used Emotional Speech Dat

_zza 4 Sep 07, 2022