An implementation for Neural Architecture Search with Random Labels (CVPR 2021 poster) on Pytorch.

Related tags

Deep LearningRLNAS
Overview

Neural Architecture Search with Random Labels(RLNAS)

Introduction

This project provides an implementation for Neural Architecture Search with Random Labels (CVPR 2021 poster) on Pytorch. Experiments are evaluated on multiple datasets (NAS-Bench-201 and ImageNet) and multiple search spaces (DARTS-like and MobileNet-like). RLNAS achieves comparable or even better results compared with state-of-the-art NAS methods such as PC-DARTS, Single Path One-Shot, even though the counterparts utilize full ground truth labels for searching. We hope our finding could inspire new understandings on the essential of NAS.

Requirements

  • Pytorch 1.4
  • Python3.5+

Search results

1.Results in NAS-Benchmark-201 search space

nas_201_results

2.Results in DARTS searh space

darts_search_sapce_results

Architeture visualization

1) Architecture searched on CIFAR-10

  • RLDARTS = Genotype(
    normal=[
    ('sep_conv_5x5', 0), ('sep_conv_3x3', 1),
    ('dil_conv_3x3', 0), ('sep_conv_5x5', 2),
    ('sep_conv_3x3', 0), ('dil_conv_5x5', 3),
    ('dil_conv_5x5', 1), ('dil_conv_3x3', 2)],
    normal_concat=[2, 3, 4, 5],
    reduce=[
    ('sep_conv_5x5', 0), ('dil_conv_3x3', 1),
    ('sep_conv_3x3', 0), ('sep_conv_5x5', 2),
    ('dil_conv_3x3', 1), ('sep_conv_3x3', 3),
    ('max_pool_3x3', 1), ('sep_conv_5x5', 2,)],
    reduce_concat=[2, 3, 4, 5])

  • Normal cell: architecture_searched_on_cifar10

  • Reduction cell: architecture_searched_on_cifar10

2) Architecture searched on ImageNet-1k without FLOPs constrain

  • RLDARTS = Genotype( normal=[
    ('sep_conv_3x3', 0), ('sep_conv_3x3', 1),
    ('sep_conv_3x3', 1), ('sep_conv_3x3', 2),
    ('sep_conv_3x3', 0), ('sep_conv_5x5', 1),
    ('sep_conv_3x3', 0), ('sep_conv_3x3', 1)],
    normal_concat=[2, 3, 4, 5],
    reduce=[
    ('sep_conv_3x3', 0), ('sep_conv_3x3', 1),
    ('sep_conv_5x5', 0), ('sep_conv_3x3', 2),
    ('sep_conv_5x5', 0), ('sep_conv_5x5', 2),
    ('sep_conv_3x3', 2), ('sep_conv_3x3', 4)],
    reduce_concat=[2, 3, 4, 5])

  • Normal cell: architecture_searched_on_imagenet_no_flops_constrain

  • Reduction cell: architecture_searched_on_cifar10

3) Architecture searched on ImageNet-1k with 600M FLOPs constrain

  • RLDARTS = Genotype(
    normal=[
    ('sep_conv_3x3', 0), ('sep_conv_3x3', 1),
    ('skip_connect', 1), ('sep_conv_3x3', 2),
    ('sep_conv_3x3', 1), ('sep_conv_3x3', 2),
    ('skip_connect', 0), ('sep_conv_3x3', 4)],
    normal_concat=[2, 3, 4, 5],
    reduce=[ ('sep_conv_3x3', 0), ('max_pool_3x3', 1),
    ('sep_conv_3x3', 0), ('skip_connect', 1),
    ('sep_conv_3x3', 0), ('dil_conv_3x3', 1),
    ('skip_connect', 0), ('sep_conv_3x3', 1)],
    reduce_concat=[2, 3, 4, 5])

  • Normal cell: architecture_searched_on_imagenet_no_flops_constrain

  • Reduction cell: architecture_searched_on_cifar10

3.Results in MobileNet search space

The MobileNet-like search space proposed in ProxylessNAS is adopted in this paper. The SuperNet contains 21 choice blocks and each block has 7 alternatives:6 MobileNet blocks (combination of kernel size {3,5,7} and expand ratio {3,6}) and ’skip-connect’.

mobilenet_search_sapce_results

Architeture visualization

mobilenet_search_sapce_results

Usage

  • RLNAS in NAS-Benchmark-201

1)enter the work directory

cd nas_bench_201

2)train supernet with random labels

bash ./scripts-search/algos/train_supernet.sh cifar10 0 1

3)evolution search with angle

bash ./scripts-search/algos/evolution_search_with_angle.sh cifar10 0 1

4)calculate correlation

bash ./scripts-search/algos/cal_correlation.sh cifar10 0 1
  • RLNAS in DARTS search space

1)enter the work directory

cd darts_search_space

search architecture on CIFAR-10

cd cifar10/rlnas/

or search architecture on ImageNet

cd imagenet/rlnas/

2)train supernet with random labels

cd train_supernet
bash run_train.sh

3)evolution search with angle

cd evolution_search
cp ../train_supernet/models/checkpoint_epoch_50.pth.tar ./model_and_data/
cp ../train_supernet/models/checkpoint_epoch_0.pth.tar ./model_and_data/
bash run_server.sh
bash run_test.sh

4)architeture evaluation

cd retrain_architetcure

add searched architecture to genotypes.py

bash run_retrain.sh
  • RLNAS in MobileNet search space

The conduct commands are almost the same steps like RLNAS in DARTS search space, excepth that you need run 'bash run_generate_flops_lookup_table.sh' before evolution search.

Note: setup a server for the distributed search

tmux new -s mq_server
sudo apt update
sudo apt install rabbitmq-server
sudo service rabbitmq-server start
sudo rabbitmqctl add_user test test
sudo rabbitmqctl set_permissions -p / test '.*' '.*' '.*'

Before search, please modify host and username in the config file evolution_search/config.py.

Citation

If you find that this project helps your research, please consider citing some of the following papers:

@article{zhang2021neural,
  title={Neural Architecture Search with Random Labels},
  author={Zhang, Xuanyang and Hou, Pengfei and Zhang, Xiangyu and Sun, Jian},
  booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
  year={2021}
}
@inproceedings{hu2020angle,
  title={Angle-based search space shrinking for neural architecture search},
  author={Hu, Yiming and Liang, Yuding and Guo, Zichao and Wan, Ruosi and Zhang, Xiangyu and Wei, Yichen and Gu, Qingyi and Sun, Jian},
  booktitle={European Conference on Computer Vision},
  pages={119--134},
  year={2020},
  organization={Springer}
}
@inproceedings{guo2020single,
  title={Single path one-shot neural architecture search with uniform sampling},
  author={Guo, Zichao and Zhang, Xiangyu and Mu, Haoyuan and Heng, Wen and Liu, Zechun and Wei, Yichen and Sun, Jian},
  booktitle={European Conference on Computer Vision},
  pages={544--560},
  year={2020},
  organization={Springer}
}
[ECCV 2020] XingGAN for Person Image Generation

Contents XingGAN or CrossingGAN Installation Dataset Preparation Generating Images Using Pretrained Model Train and Test New Models Evaluation Acknowl

Hao Tang 218 Oct 29, 2022
yolox_backbone is a deep-learning library and is a collection of YOLOX Backbone models.

YOLOX-Backbone yolox-backbone is a deep-learning library and is a collection of YOLOX backbone models. Install pip install yolox-backbone Load a Pret

Yonghye Kwon 21 Dec 28, 2022
The implementation our EMNLP 2021 paper "Enhanced Language Representation with Label Knowledge for Span Extraction".

LEAR The implementation our EMNLP 2021 paper "Enhanced Language Representation with Label Knowledge for Span Extraction". See below for an overview of

杨攀 93 Jan 07, 2023
A python library to build Model Trees with Linear Models at the leaves.

A python library to build Model Trees with Linear Models at the leaves.

Marco Cerliani 212 Dec 30, 2022
VIsually-Pivoted Audio and(N) Text

VIP-ANT: VIsually-Pivoted Audio and(N) Text Code for the paper Connecting the Dots between Audio and Text without Parallel Data through Visual Knowled

Yän.PnG 16 Nov 04, 2022
SNIPS: Solving Noisy Inverse Problems Stochastically

SNIPS: Solving Noisy Inverse Problems Stochastically This repo contains the official implementation for the paper SNIPS: Solving Noisy Inverse Problem

Bahjat Kawar 35 Nov 09, 2022
BRNet - code for Automated assessment of BI-RADS categories for ultrasound images using multi-scale neural networks with an order-constrained loss function

BRNet code for "Automated assessment of BI-RADS categories for ultrasound images using multi-scale neural networks with an order-constrained loss func

Yong Pi 2 Mar 09, 2022
A PaddlePaddle implementation of STGCN with a few modifications in the model architecture in order to forecast traffic jam.

About This repository contains the code of a PaddlePaddle implementation of STGCN based on the paper Spatio-Temporal Graph Convolutional Networks: A D

Tianjian Li 1 Jan 11, 2022
Relative Uncertainty Learning for Facial Expression Recognition

Relative Uncertainty Learning for Facial Expression Recognition The official implementation of the following paper at NeurIPS2021: Title: Relative Unc

35 Dec 28, 2022
QuanTaichi evaluation suite

QuanTaichi: A Compiler for Quantized Simulations (SIGGRAPH 2021) Yuanming Hu, Jiafeng Liu, Xuanda Yang, Mingkuan Xu, Ye Kuang, Weiwei Xu, Qiang Dai, W

Taichi Developers 120 Jan 04, 2023
Federated_learning codes used for the the paper "Evaluation of Federated Learning Aggregation Algorithms" and "A Federated Learning Aggregation Algorithm for Pervasive Computing: Evaluation and Comparison"

Federated Distance (FedDist) This is the code accompanying the Percom2021 paper "A Federated Learning Aggregation Algorithm for Pervasive Computing: E

GETALP 8 Jan 03, 2023
A python script to convert images to animated sus among us crewmate twerk jifs as seen on r/196

img_sussifier A python script to convert images to animated sus among us crewmate twerk jifs as seen on r/196 Examples How to use install python pip i

41 Sep 30, 2022
learned_optimization: Training and evaluating learned optimizers in JAX

learned_optimization: Training and evaluating learned optimizers in JAX learned_optimization is a research codebase for training learned optimizers. I

Google 533 Dec 30, 2022
A python library to artfully visualize Factorio Blueprints and an interactive web demo for using it.

Factorio Blueprint Visualizer I love the game Factorio and I really like the look of factories after growing for many hours or blueprints after tweaki

Piet Brömmel 124 Jan 07, 2023
Full-featured Decision Trees and Random Forests learner.

CID3 This is a full-featured Decision Trees and Random Forests learner. It can save trees or forests to disk for later use. It is possible to query tr

Alejandro Penate-Diaz 3 Aug 15, 2022
FEMDA: Robust classification with Flexible Discriminant Analysis in heterogeneous data

FEMDA: Robust classification with Flexible Discriminant Analysis in heterogeneous data. Flexible EM-Inspired Discriminant Analysis is a robust supervised classification algorithm that performs well i

0 Sep 06, 2022
SCALE: Modeling Clothed Humans with a Surface Codec of Articulated Local Elements (CVPR 2021)

SCALE: Modeling Clothed Humans with a Surface Codec of Articulated Local Elements (CVPR 2021) This repository contains the official PyTorch implementa

Qianli Ma 133 Jan 05, 2023
SiT: Self-supervised vIsion Transformer

This repository contains the official PyTorch self-supervised pretraining, finetuning, and evaluation codes for SiT (Self-supervised image Transformer).

Sara Ahmed 275 Dec 28, 2022
Setup and customize deep learning environment in seconds.

Deepo is a series of Docker images that allows you to quickly set up your deep learning research environment supports almost all commonly used deep le

Ming 6.3k Jan 06, 2023
Deep Learning for Time Series Forecasting.

nixtlats:Deep Learning for Time Series Forecasting [nikstla] (noun, nahuatl) Period of time. State-of-the-art time series forecasting for pytorch. Nix

Nixtla 5 Dec 06, 2022