NAS-HPO-Bench-II is the first benchmark dataset for joint optimization of CNN and training HPs.

Overview

NAS-HPO-Bench-II API

Overview

NAS-HPO-Bench-II is the first benchmark dataset for joint optimization of CNN and training HPs.

It helps

  • a fair and low-cost evaluation/comparison of joint optimization (NAS+HPO) methods
  • a detailed analysis of the relationship between architecture/training HPs and performances

Our experimental analysis supports the importance of joint optimization. Please see our paper for details.

This repo provides API for NAS-HPO-Bench-II to make benchmarking easy. You can query our data when evaluating models in the search process of AutoML methods instead of training the models at a high cost.

If you use the dataset, please cite:

@InProceedings{hirose2021bench,
  title={{NAS-HPO-Bench-II}: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters},
  author={Hirose, Yoichi and Yoshinari, Nozomu and Shirakawa,  Shinichi},
  booktitle={Proceedings of the 13th Asian Conference on Machine Learning},
  year={2021}
}

The code for training models is here.

Dataset Overview

The total size of the search space is 192K. The dataset includes

  • the exact data of all the models in the search space for 12 epoch training
  • the surrogate data predicting accuracies after 200 epoch training

Architecture Search Space

The overall CNN architecture is constructed by stacking cells represented as a directed acyclic graph (DAG). Each edge in the graph indicates one of the four operations.

  • 3x3 convolution (ReLU activation, 3x3 convolution with stride 1, then batch normalization)
  • 3x3 average pooling with stride 1
  • Skip, which outputs the input tensor
  • Zero, which outputs the zero tensor with the same dimension as the input

It is based on NAS-Bench-201 and the only difference is that we exclude the 1x1 convolution operation from the options.

Training HP Search Space

The combination of eight initial learning rates and six batch sizes are used.

Hyperparameter Options
Batch Size 16, 32, 64, 128, 256, 512
Learning Rate 0.003125, 0.00625, 0.0125, 0.025, 0.05, 0.1, 0.2, 0.4

Installation

Run

pip install nashpobench2api

, and download the API dataset from Google Drive (93.7MB), then put the data in some directory (default: ./data). This API supports python >= 3.6 (and no external library dependencies).

If you want to run the codes in bench_algos, run pip install -r requirements.txt.

Getting Started

Create an API instance to get access to the dataset.

from nashpobench2api import NASHPOBench2API as API
api = API('/path/to/dataset')

You can get 12-epoch valid accuracy (%) and train+valid training cost (sec.) of the specified configuration.

acc, cost = api.query_by_key(
	cellcode='0|10|210',
	batch_size=256,
	lr=0.1 )

Here, cellcode represents one of the architectures in the search space. As shown in the figure below, the numbers in the cellcode mean the type of operations, and the position of the numbers shows the edge '(A) | (B)(C) | (D)(E)(F)'.

In the querying process, the api instance remembers and shows the log (what you have queried). You can reduce the log if set verbose=False when initializing api.

When the querying process has finished, you can get the test accuracy of the configuration with the best valid accuracy in the queried configurations.

results = api.get_results()

results is a dictionary with the keys below.

Key Explanation
acc_trans a transition of valid accuracies api have queried
key_trans a transition of keys (=cellcode, lr, batch_size) api have queried
best_acc_trans a transition of the best valid accuracies (%) api have queried
best_key_trans a transition of the best keys (=cellcode, lr, batch_size) api have queried
total_cost_trans a transition of train+valid costs (sec.)
final_accs 12-epoch and 200-epoch test accuracies (%) of the key with the best valid accuracy api have queried

You can reset what api have remebered, which is useful when multiple runs.

api.reset_log_data()

The examples of benchmarking codes are in the bench_algos directory. Especially, random_search.py is the simplest code and easy to understand (the core part is random_search()).

Work in Progress

  • Upload the dataset as DataFrame for visualization/analysis.
  • Upload codes for a surrogate model.
  • Upload the trained models.
Owner
yoichi hirose
yoichi hirose
💊 A 3D Generative Model for Structure-Based Drug Design (NeurIPS 2021)

A 3D Generative Model for Structure-Based Drug Design Coming soon... Citation @inproceedings{luo2021sbdd, title={A 3D Generative Model for Structu

Shitong Luo 118 Jan 05, 2023
Code for LIGA-Stereo Detector, ICCV'21

LIGA-Stereo Introduction This is the official implementation of the paper LIGA-Stereo: Learning LiDAR Geometry Aware Representations for Stereo-based

Xiaoyang Guo 75 Dec 09, 2022
Implementing yolov4 target detection and tracking based on nao robot

Implementing yolov4 target detection and tracking based on nao robot

6 Apr 19, 2022
Causal-BALD: Deep Bayesian Active Learning of Outcomes to Infer Treatment-Effects from Observational Data.

causal-bald | Abstract | Installation | Example | Citation | Reproducing Results DUE An implementation of the methods presented in Causal-BALD: Deep B

OATML 13 Oct 07, 2022
A large-scale face dataset for face parsing, recognition, generation and editing.

CelebAMask-HQ [Paper] [Demo] CelebAMask-HQ is a large-scale face image dataset that has 30,000 high-resolution face images selected from the CelebA da

switchnorm 1.7k Dec 26, 2022
Temporally Efficient Vision Transformer for Video Instance Segmentation, CVPR 2022, Oral

Temporally Efficient Vision Transformer for Video Instance Segmentation Temporally Efficient Vision Transformer for Video Instance Segmentation (CVPR

Hust Visual Learning Team 203 Dec 31, 2022
Image Super-Resolution by Neural Texture Transfer

SRNTT: Image Super-Resolution by Neural Texture Transfer Tensorflow implementation of the paper Image Super-Resolution by Neural Texture Transfer acce

Zhifei Zhang 413 Nov 30, 2022
Learning with Subset Stacking

Learning with Subset Stacking (LESS) LESS is a new supervised learning algorithm that is based on training many local estimators on subsets of a given

S. Ilker Birbil 19 Oct 04, 2022
Stacs-ci - A set of modules to enable integration of STACS with commonly used CI / CD systems

Static Token And Credential Scanner CI Integrations What is it? STACS is a YARA

STACS 18 Aug 04, 2022
Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations, CVPR 2019 (Oral)

Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations The code of: Weakly Supervised Learning of Instance Segmentation with I

Jiwoon Ahn 472 Dec 29, 2022
Attack classification models with transferability, black-box attack; unrestricted adversarial attacks on imagenet

Attack classification models with transferability, black-box attack; unrestricted adversarial attacks on imagenet, CVPR2021 安全AI挑战者计划第六期:ImageNet无限制对抗攻击 决赛第四名(team name: Advers)

51 Dec 01, 2022
Aspect-Sentiment-Multiple-Opinion Triplet Extraction (NLPCC 2021)

The code and data for the paper "Aspect-Sentiment-Multiple-Opinion Triplet Extraction" Requirements Python 3.6.8 torch==1.2.0 pytorch-transformers==1.

慢半拍 5 Jul 02, 2022
This repository is a basic Machine Learning train & validation Template (Using PyTorch)

pytorch_ml_template This repository is a basic Machine Learning train & validation Template (Using PyTorch) TODO Markdown 사용법 Build Docker 사용법 Anacond

1 Sep 15, 2022
[Preprint] "Chasing Sparsity in Vision Transformers: An End-to-End Exploration" by Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, Zhangyang Wang

Chasing Sparsity in Vision Transformers: An End-to-End Exploration Codes for [Preprint] Chasing Sparsity in Vision Transformers: An End-to-End Explora

VITA 64 Dec 08, 2022
Attention Probe: Vision Transformer Distillation in the Wild

Attention Probe: Vision Transformer Distillation in the Wild Jiahao Wang, Mingdeng Cao, Shuwei Shi, Baoyuan Wu, Yujiu Yang In ICASSP 2022 This code is

Wang jiahao 3 Oct 31, 2022
PyTorch module to use OpenFace's nn4.small2.v1.t7 model

OpenFace for Pytorch Disclaimer: This codes require the input face-images that are aligned and cropped in the same way of the original OpenFace. * I m

Pete Tae-hoon Kim 176 Dec 12, 2022
A Tensorflow implementation of the Text Conditioned Auxiliary Classifier Generative Adversarial Network for Generating Images from text descriptions

A Tensorflow implementation of the Text Conditioned Auxiliary Classifier Generative Adversarial Network for Generating Images from text descriptions

Ayushman Dash 93 Aug 04, 2022
Code for SentiBERT: A Transferable Transformer-Based Architecture for Compositional Sentiment Semantics (ACL'2020).

SentiBERT Code for SentiBERT: A Transferable Transformer-Based Architecture for Compositional Sentiment Semantics (ACL'2020). https://arxiv.org/abs/20

Da Yin 66 Aug 13, 2022
UFPR-ADMR-v2 Dataset

UFPR-ADMR-v2 Dataset The UFPR-ADMRv2 dataset contains 5,000 dial meter images obtained on-site by employees of the Energy Company of Paraná (Copel), w

Gabriel Salomon 8 Sep 29, 2022
Code for "Causal autoregressive flows" - AISTATS, 2021

Code for "Causal Autoregressive Flow" This repository contains code to run and reproduce experiments presented in Causal Autoregressive Flows, present

Ricardo Pio Monti 35 Dec 16, 2022