DeepOBS: A Deep Learning Optimizer Benchmark Suite

Related tags

Deep LearningDeepOBS
Overview

DeepOBS - A Deep Learning Optimizer Benchmark Suite

DeepOBS

PyPI version Documentation Status License: MIT

DeepOBS is a benchmarking suite that drastically simplifies, automates and improves the evaluation of deep learning optimizers.

It can evaluate the performance of new optimizers on a variety of real-world test problems and automatically compare them with realistic baselines.

DeepOBS automates several steps when benchmarking deep learning optimizers:

  • Downloading and preparing data sets.
  • Setting up test problems consisting of contemporary data sets and realistic deep learning architectures.
  • Running the optimizers on multiple test problems and logging relevant metrics.
  • Reporting and visualization the results of the optimizer benchmark.

DeepOBS Output

This branch contains the beta of version 1.2.0 with TensorFlow and PyTorch support. It is currently in a pre-release state. Not all features are implemented and most notably we currently don't provide baselines for this version.

The full documentation of this beta version is available on readthedocs: https://deepobs-with-pytorch.readthedocs.io/

The paper describing DeepOBS has been accepted for ICLR 2019 and can be found here: https://openreview.net/forum?id=rJg6ssC5Y7

If you find any bugs in DeepOBS, or find it hard to use, please let us know. We are always interested in feedback and ways to improve DeepOBS.

Installation

pip install -e git+https://github.com/fsschneider/[email protected]#egg=DeepOBS

We tested the package with Python 3.6, TensorFlow version 1.12, Torch version 1.1.0 and Torchvision version 0.3.0. Other versions might work, and we plan to expand compatibility in the future.

Further tutorials and a suggested protocol for benchmarking deep learning optimizers can be found on https://deepobs-with-pytorch.readthedocs.io/

Comments
  • Request: Share the hyper-parameters found in the grid search

    Request: Share the hyper-parameters found in the grid search

    To lessen the burden of re-running the benchmark, would it be possible to publish the optimal hyper-parameters somewhere?

    By-reusing those hyper-parameters, one would avoid the most computationally-demanding part of reproducing the results (by 1-2 orders of magnitude).

    opened by jotaf98 2
  • Add functionality to skip existing runs, plotting modes, some refactoring

    Add functionality to skip existing runs, plotting modes, some refactoring

    • Adding parameter skip_if_exists to runner.run
      • Default value is set such that the current behavior is maintained
      • By setting to True, runs that already have a .json output file will not be executed again
    • Possible extensions
      • Make skip_if_exists arg-parsable
    opened by f-dangel 2
  • KeyError: 'optimizer_hyperparams'

    KeyError: 'optimizer_hyperparams'

    (Apologies for creating multiple issues in a row -- it seemed more clean to keep them separate.)

    I downloaded the data from DeepOBS_Baselines, and attempted to run example_analyze_pytorch.py. Unfortunately DeepOBS seems to look for keys in the JSON files that don't exist:

    $ python example_analyze_pytorch.py
    /users/user/Research/deepobs/deepobs/analyzer/shared_utils.py:144: RuntimeWarning: Metric valid_accu
    racies does not exist for testproblem quadratic_deep. We now use fallback metric valid_losses
      default_metric), RuntimeWarning)
    /users/user/Research/deepobs/deepobs/analyzer/shared_utils.py:229: RuntimeWarning: All settings for
    /scratch/local/ssd/user/data/deepobs/quadratic_deep/SGD on test problem quadratic_deep have the same
     number of seeds runs. Mode 'most' does not make sense and we use the fallback mode 'final'
      .format(optimizer_path, testproblem_name), RuntimeWarning)
    {'Performance': 127.96759578159877, 'Speed': 'N.A.', 'Hyperparameters': {'lr': 0.01, 'momentum': 0.9
    9, 'nesterov': False}, 'Training Parameters': {}}
    /users/user/Research/deepobs/deepobs/analyzer/shared_utils.py:144: RuntimeWarning: Metric valid_accu
    racies does not exist for testproblem quadratic_deep. We now use fallback metric valid_losses
      default_metric), RuntimeWarning)
    /users/user/Research/deepobs/deepobs/analyzer/shared_utils.py:229: RuntimeWarning: All settings for
    /scratch/local/ssd/user/data/deepobs/quadratic_deep/SGD on test problem quadratic_deep have the same
     number of seeds runs. Mode 'most' does not make sense and we use the fallback mode 'final'
      .format(optimizer_path, testproblem_name), RuntimeWarning)
    /users/user/Research/deepobs/deepobs/analyzer/shared_utils.py:150: RuntimeWarning: Cannot fallback t
    o metric valid_losses for optimizer MomentumOptimizer on testproblem quadratic_deep. Will now fallba
    ck to metric test_losses
      testproblem_name), RuntimeWarning)
    /users/user/miniconda3/lib/python3.7/site-packages/numpy/core/_methods.py:193: RuntimeWarning: inva$
    id value encountered in subtract
      x = asanyarray(arr - arrmean)
    /users/user/miniconda3/lib/python3.7/site-packages/numpy/lib/function_base.py:3949: RuntimeWarning:
    invalid value encountered in multiply
      x2 = take(ap, indices_above, axis=axis) * weights_above
    Traceback (most recent call last):
      File "example_analyze_pytorch.py", line 17, in <module>
        analyzer.plot_optimizer_performance(result_path, reference_path=base + '/deepobs/baselines/quad$
    atic_deep/MomentumOptimizer')
      File "/users/user/Research/deepobs/deepobs/analyzer/analyze.py", line 514, in plot_optimizer_perfo
    rmance
        which=which)
      File "/users/user/Research/deepobs/deepobs/analyzer/analyze.py", line 462, in _plot_optimizer_perf
    ormance
        optimizer_path, mode, metric)
      File "/users/user/Research/deepobs/deepobs/analyzer/shared_utils.py", line 206, in create_setting_
    analyzer_ranking
        setting_analyzers = _get_all_setting_analyzer(optimizer_path)
      File "/users/user/Research/deepobs/deepobs/analyzer/shared_utils.py", line 184, in _get_all_settin
    g_analyzer
        setting_analyzers.append(SettingAnalyzer(sett_path))
      File "/users/user/Research/deepobs/deepobs/analyzer/shared_utils.py", line 260, in __init__
        self.aggregate = aggregate_runs(path)
      File "/users/user/Research/deepobs/deepobs/analyzer/shared_utils.py", line 101, in aggregate_runs
        aggregate['optimizer_hyperparams'] = json_data['optimizer_hyperparams']
    KeyError: 'optimizer_hyperparams'
    

    One of the JSON files in question looks like this (data points snipped for brevity):

    {
    "train_losses": [353.9337594168527, 347.5994306291853, 331.35902622767856, 307.2468915666853, ... 97.28871154785156, 91.45470428466797, 96.45774841308594, 86.27237701416016],
    "optimizer": "MomentumOptimizer",
    "testproblem": "quadratic_deep",
    "weight_decay": null,
    "batch_size": 128,
    "num_epochs": 100,
    "learning_rate": 1e-05,
    "lr_sched_epochs": null,
    "lr_sched_factors": null,
    "random_seed": 42,
    "train_log_interval": 1,
    "hyperparams": {"momentum": 0.99, "use_nesterov": false}
    }
    

    The obvious key seems to be hyperparams as opposed to optimizer_hyperparams; this occurs only for some JSON files.

    Edit: Having fixed this, there is a further key error on training_params. Perhaps these were generated with different versions of the package.

    opened by jotaf98 3
  • Installation error / unmentioned dependency

    Installation error / unmentioned dependency "bayes_opt"

    Attempting to install by following the documentation's instructions, after installing all the mentioned dependencies with conda, results in the following error:

    (base) [email protected]:~$ pip install -e git+https://github.com/abahde/[email protected]#egg=DeepOBS
    Obtaining DeepOBS from git+https://github.com/abahde/[email protected]#egg=DeepOBS
      Cloning https://github.com/abahde/DeepOBS.git (to revision master) to ./src/deepobs
      Running command git clone -q https://github.com/abahde/DeepOBS.git /users/user/src/deepobs
        ERROR: Complete output from command python setup.py egg_info:
        ERROR: Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "/users/user/src/deepobs/setup.py", line 5, in <module>
            from deepobs import __version__
          File "/users/user/src/deepobs/deepobs/__init__.py", line 5, in <module>
            from . import analyzer
          File "/users/user/src/deepobs/deepobs/analyzer/__init__.py", line 2, in <module>
            from . import analyze
          File "/users/user/src/deepobs/deepobs/analyzer/analyze.py", line 12, in <module>
            from ..tuner.tuner_utils import generate_tuning_summary
          File "/users/user/src/deepobs/deepobs/tuner/__init__.py", line 4, in <module>
            from .bayesian import GP
          File "/users/user/src/deepobs/deepobs/tuner/bayesian.py", line 3, in <module>
            from bayes_opt import UtilityFunction
        ModuleNotFoundError: No module named 'bayes_opt'
        ----------------------------------------
    ERROR: Command "python setup.py egg_info" failed with error code 1 in /users/user/src/deepobs/
    

    Is this bayes_opt package really necessary? It seems a bit tangential to the package's purpose (or at most optional).

    Edit: It turns out that bayesian-optimization has relatively few requirements so this is not a big issue; perhaps just the docs need updating.

    As an aside, it might be possible to suggest a single conda command that installs everything: conda install -c conda-forge seaborn matplotlib2tikz bayesian-optimization.

    opened by jotaf98 0
  • Wall-clock time plots

    Wall-clock time plots

    Optimizers can have very different runtimes per iteration, especially 2nd-order ones.

    This means that sometimes, despite promises of "faster" convergence, the wall-clock time taken to converge is disappointingly larger.

    Is there any chance DeepOBS could implement wall-clock time plots, in addition to per-epoch ones? (E.g. X axis in minutes or hours.)

    opened by jotaf98 4
  • Improve estimate_runtime()

    Improve estimate_runtime()

    There are a couple of improvements that I suggest:

    • [ ] Return the results not as a string, but as a dict or an object.
    • [ ] (Maybe, think about that) Include the ability to test multiple optimizers simultaneously.
    • [ ] Report standard deviation and individual runtimes for SGD.
    • [ ] Add a function that generates a figure, similar to https://github.com/ludwigbald/probprec/blob/master/code/exp_perf_prec/analyze.py
    opened by ludwigbald 0
  • Implement validation set split also for TensorFlow

    Implement validation set split also for TensorFlow

    In PyTorch we split the validation set from the training set randomly. It has the size of the test set. The validation performance is used by the tuner and analyzer to obtain the best instance. This split should be implemented in the TensorFlow data sets as well. We have already prepared the test problem and the runner implementations for this change. The only change that needs to be done to the runner is marked in the code with a ToDo flag.

    bug enhancement 
    opened by abahde 0
Releases(v1.2.0-beta)
  • v1.2.0-beta(Sep 17, 2019)

    Draft of release notes:

    • A PyTorch implementation (though not for all test problems yet)
    • A refactored Analyzer module (more flexibility and interpretability)
    • A Tuning module that automates the tuning process
    • Some minor improvements of the TensorFlow code (important bugfix: fmnist_mlp now really uses F-MNIST and not MNIST)
    • For the PyTorch code a validation set metric for each test problem. However, so far, the TensorFlow code comes without validation sets.
    • Runners now break from training if the loss becomes NaN.
    • Runners now return the output dictionary.
    • Additional training parameters can be passed as kwargs to the run() method.
    • Numpy is now also seeded.
    • Small and large benchmark sets are now global variables in DeepOBS.
    • Default test problem settings are now a global variable in DeepOBS.
    • JSON output is now dumped in human readable format.
    • Accuracy is now only printed if available.
    • Simplified Runner API.
    • Learning Rate Schedule Runner is now an extra class.
    Source code(tar.gz)
    Source code(zip)
Owner
Aaron Bahde
Graduate student at the University of Tübingen, Methods of Machine Learning
Aaron Bahde
[ICCV'21] Official implementation for the paper Social NCE: Contrastive Learning of Socially-aware Motion Representations

CrowdNav with Social-NCE This is an official implementation for the paper Social NCE: Contrastive Learning of Socially-aware Motion Representations by

VITA lab at EPFL 125 Dec 23, 2022
The world's simplest facial recognition api for Python and the command line

Face Recognition You can also read a translated version of this file in Chinese 简体中文版 or in Korean 한국어 or in Japanese 日本語. Recognize and manipulate fa

Adam Geitgey 46.9k Jan 03, 2023
Code for the paper "M2m: Imbalanced Classification via Major-to-minor Translation" (CVPR 2020)

M2m: Imbalanced Classification via Major-to-minor Translation This repository contains code for the paper "M2m: Imbalanced Classification via Major-to

79 Oct 13, 2022
Taking A Closer Look at Domain Shift: Category-level Adversaries for Semantics Consistent Domain Adaptation

Taking A Closer Look at Domain Shift: Category-level Adversaries for Semantics Consistent Domain Adaptation (CVPR2019) This is a pytorch implementatio

Yawei Luo 280 Jan 01, 2023
[CVPR 2021 Oral] ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis

ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis [arxiv|pdf|v

Yinan He 78 Dec 22, 2022
Package for extracting emotions from social media text. Tailored for financial data.

EmTract: Extracting Emotions from Social Media Text Tailored for Financial Contexts EmTract is a tool that extracts emotions from social media text. I

13 Nov 17, 2022
GANfolk: Using AI to create portraits of fictional people to sell as NFTs

GANfolk are AI-generated renderings of fictional people. Each image in the collection was created by a pair of Generative Adversarial Networks (GANs) with names and backstories also created with AI.

Robert A. Gonsalves 32 Dec 02, 2022
PyTorch implementation for Convolutional Networks with Adaptive Inference Graphs

Convolutional Networks with Adaptive Inference Graphs (ConvNet-AIG) This repository contains a PyTorch implementation of the paper Convolutional Netwo

Andreas Veit 176 Dec 07, 2022
PipeTransformer: Automated Elastic Pipelining for Distributed Training of Large-scale Models

PipeTransformer: Automated Elastic Pipelining for Distributed Training of Large-scale Models This repository is the official implementation of the fol

DistributedML 41 Dec 06, 2022
I will implement Fastai in each projects present in this repository.

DEEP LEARNING FOR CODERS WITH FASTAI AND PYTORCH The repository contains a list of the projects which I have worked on while reading the book Deep Lea

Thinam Tamang 43 Dec 20, 2022
Official Codes for Graph Modularity:Towards Understanding the Cross-Layer Transition of Feature Representations in Deep Neural Networks.

Dynamic-Graphs-Construction Official Codes for Graph Modularity:Towards Understanding the Cross-Layer Transition of Feature Representations in Deep Ne

11 Dec 14, 2022
Official PyTorch Implementation for InfoSwap: Information Bottleneck Disentanglement for Identity Swapping

InfoSwap: Information Bottleneck Disentanglement for Identity Swapping Code usage Please check out the user manual page. Paper Gege Gao, Huaibo Huang,

Grace Hešeri 56 Dec 20, 2022
Binary classification for arrythmia detection with ECG datasets.

HEART DISEASE AI DATATHON 2021 [Eng] / [Kor] #English This is an AI diagnosis modeling contest that uses the heart disease echocardiography and electr

HY_Kim 3 Jul 14, 2022
Torch implementation of SegNet and deconvolutional network

Torch implementation of SegNet and deconvolutional network

Fedor Chervinskii 5 Jul 17, 2020
Implementation of Gans

GAN Generative Adverserial Networks are an approach to generative data modelling using Deep learning methods. I have currently implemented : DCGAN on

Sibam Parida 5 Sep 07, 2021
Statistical and Algorithmic Investing Strategies for Everyone

Eiten - Algorithmic Investing Strategies for Everyone Eiten is an open source toolkit by Tradytics that implements various statistical and algorithmic

Tradytics 2.5k Jan 02, 2023
Demonstration of the Model Training as a CI/CD System in Vertex AI

Model Training as a CI/CD System This project demonstrates the machine model training as a CI/CD system in GCP platform. You will see more detailed wo

Chansung Park 19 Dec 28, 2022
BOVText: A Large-Scale, Multidimensional Multilingual Dataset for Video Text Spotting

BOVText: A Large-Scale, Bilingual Open World Dataset for Video Text Spotting Updated on December 10, 2021 (Release all dataset(2021 videos)) Updated o

weijiawu 47 Dec 26, 2022
A Large Scale Benchmark for Individual Treatment Effect Prediction and Uplift Modeling

large-scale-ITE-UM-benchmark This repository contains code and data to reproduce the results of the paper "A Large Scale Benchmark for Individual Trea

10 Nov 19, 2022
Transformer part of 12th place solution in Riiid! Answer Correctness Prediction

kaggle_riiid Transformer part of 12th place solution in Riiid! Answer Correctness Prediction. Please see here for more information. Execution You need

Sakami Kosuke 2 Apr 23, 2022