A Lightweight Hyperparameter Optimization Tool ๐Ÿš€

Overview

Lightweight Hyperparameter Optimization ๐Ÿš€

Pyversions PyPI version Code style: black Colab

The mle-hyperopt package provides a simple and intuitive API for hyperparameter optimization of your Machine Learning Experiment (MLE) pipeline. It supports real, integer & categorical search variables and single- or multi-objective optimization.

Core features include the following:

  • API Simplicity: strategy.ask(), strategy.tell() interface & space definition.
  • Strategy Diversity: Grid, random, coordinate search, SMBO & wrapping around FAIR's nevergrad.
  • Search Space Refinement based on the top performing configs via strategy.refine(top_k=10).
  • Export of configurations to execute via e.g. python train.py --config_fname config.yaml.
  • Storage & reload search logs via strategy.save(<log_fname>), strategy.load(<log_fname>).

For a quickstart check out the notebook blog ๐Ÿ“– .

The API ๐ŸŽฎ

from mle_hyperopt import RandomSearch

# Instantiate random search class
strategy = RandomSearch(real={"lrate": {"begin": 0.1,
                                        "end": 0.5,
                                        "prior": "log-uniform"}},
                        integer={"batch_size": {"begin": 32,
                                                "end": 128,
                                                "prior": "uniform"}},
                        categorical={"arch": ["mlp", "cnn"]})

# Simple ask - eval - tell API
configs = strategy.ask(5)
values = [train_network(**c) for c in configs]
strategy.tell(configs, values)

Implemented Search Types ๐Ÿ”ญ

Search Type Description search_config
drawing GridSearch Search over list of discrete values -
drawing RandomSearch Random search over variable ranges refine_after, refine_top_k
drawing CoordinateSearch Coordinate-wise optimization with fixed defaults order, defaults
drawing SMBOSearch Sequential model-based optimization base_estimator, acq_function, n_initial_points
drawing NevergradSearch Multi-objective nevergrad wrapper optimizer, budget_size, num_workers

Variable Types & Hyperparameter Spaces ๐ŸŒ

Variable Type Space Specification
drawing real Real-valued Dict: begin, end, prior/bins (grid)
drawing integer Integer-valued Dict: begin, end, prior/bins (grid)
drawing categorical Categorical List: Values to search over

Installation โณ

A PyPI installation is available via:

pip install mle-hyperopt

Alternatively, you can clone this repository and afterwards 'manually' install it:

git clone https://github.com/mle-infrastructure/mle-hyperopt.git
cd mle-hyperopt
pip install -e .

Further Options ๐Ÿšด

Saving & Reloading Logs ๐Ÿช

# Storing & reloading of results from .pkl
strategy.save("search_log.json")
strategy = RandomSearch(..., reload_path="search_log.json")

# Or manually add info after class instantiation
strategy = RandomSearch(...)
strategy.load("search_log.json")

Search Decorator ๐Ÿงถ

from mle_hyperopt import hyperopt

@hyperopt(strategy_type="grid",
          num_search_iters=25,
          real={"x": {"begin": 0., "end": 0.5, "bins": 5},
                "y": {"begin": 0, "end": 0.5, "bins": 5}})
def circle(config):
    distance = abs((config["x"] ** 2 + config["y"] ** 2))
    return distance

strategy = circle()

Storing Configuration Files ๐Ÿ“‘

# Store 2 proposed configurations - eval_0.yaml, eval_1.yaml
strategy.ask(2, store=True)
# Store with explicit configuration filenames - conf_0.yaml, conf_1.yaml
strategy.ask(2, store=True, config_fnames=["conf_0.yaml", "conf_1.yaml"])

Retrieving Top Performers & Visualizing Results ๐Ÿ“‰

# Get the top k best performing configurations
id, configs, values = strategy.get_best(top_k=4)

# Plot timeseries of best performing score over search iterations
strategy.plot_best()

# Print out ranking of best performers
strategy.print_ranking(top_k=3)

Refining the Search Space of Your Strategy ๐Ÿช“

# Refine the search space after 5 & 10 iterations based on top 2 configurations
strategy = RandomSearch(real={"lrate": {"begin": 0.1,
                                        "end": 0.5,
                                        "prior": "log-uniform"}},
                        integer={"batch_size": {"begin": 1,
                                                "end": 5,
                                                "prior": "uniform"}},
                        categorical={"arch": ["mlp", "cnn"]},
                        search_config={"refine_after": [5, 10],
                                       "refine_top_k": 2})

# Or do so manually using `refine` method
strategy.tell(...)
strategy.refine(top_k=2)

Note that the search space refinement is only implemented for random, SMBO and nevergrad-based search strategies.

Development & Milestones for Next Release

You can run the test suite via python -m pytest -vv tests/. If you find a bug or are missing your favourite feature, feel free to contact me @RobertTLange or create an issue ๐Ÿค— .

  • Robust type checking with isinstance(self.log[0]["objective"], (float, int, np.integer, np.float))
  • Add improvement method indicating if score is better than best stored one
  • Fix logging message when log is stored
  • Add save option for best plot
  • Make json serializer more robust for numpy data types
  • Make sure search space refinement works for different batch sizes
  • Add args, kwargs into decorator
  • Check why SMBO can propose same config multiple times. Add Hutter reference.
Comments
  • [FEATURE] Hyperband

    [FEATURE] Hyperband

    Hi! I was wondering if the Hyperband hyperparameter algorithm is something you want implemented.

    I'm willing to spend some time working on it if there's interest.

    opened by colligant 5
  • [FEATURE] Option to pickle the whole strategy

    [FEATURE] Option to pickle the whole strategy

    Right now strategy.save produces a JSON with the log. Any reason you didn't opt for (or have an option of) pickling the whole strategy? Two motivations for this:

    1. Not having to re-init the strategy with all the args/kwargs
    2. Not having to loop through tell! SMBO can take quite some time to do this.
    opened by alexander-soare 4
  • Type checking strategy.log could be made more flexible?

    Type checking strategy.log could be made more flexible?

    Yay first issue! Congrats Robert, this is a great interface. Haven't used a hyperopt library in a while and this felt so easy to pick up.


    For example https://github.com/RobertTLange/mle-hyperopt/blob/57eb806e95c854f48f8faac2b2dc182d2180d393/mle_hyperopt/search.py#L251

    raises an error if my objective is numpy.float64. Also noticed https://github.com/RobertTLange/mle-hyperopt/blob/57eb806e95c854f48f8faac2b2dc182d2180d393/mle_hyperopt/search.py#L206

    Could we just have

    isinstance(strategy.log[0]['objective'], (float, int))
    

    which would cover the numpy types?

    opened by alexander-soare 4
  • Successive Halving, Hyperband, PBT

    Successive Halving, Hyperband, PBT

    • [x] Robust type checking with isinstance(self.log[0]["objective"], (float, int, np.integer, np.float))
    • [x] Add improvement method indicating if score is better than best stored one
    • [x] Fix logging message when log is stored
    • [x] Add save option for best plot
    • [x] Make json serializer more robust for numpy data types
    • [x] Add possibility to save as .pkl file by providing filename in .save method ending with .pkl (issue #2)
    • [x] Add args, kwargs into decorator
    • [x] Adds synchronous Successive Halving (SuccessiveHalvingSearch - issue #3)
    • [x] Adds synchronous HyperBand (HyperbandSearch - issue #3)
    • [x] Adds synchronous PBT (PBTSearch - issue #4 )
    opened by RobertTLange 1
  • [Feature] Synchronous PBT

    [Feature] Synchronous PBT

    Move PBT ask/tell functionality from mle-toolbox experimental to mle-hyperopt. Is there any literature/empirical evidence for the importance of being asynchronous?

    enhancement 
    opened by RobertTLange 1
Releases(v0.0.7)
  • v0.0.7(Feb 20, 2022)

    Added

    • Log reloading helper for post-processing.

    Fixed

    • Bug fix in mle-search with imports of dependencies. Needed to append path.
    • Bug fix with cleaning nested dictionaries. Have to make sure not to delete entire sub-dictionary.
    Source code(tar.gz)
    Source code(zip)
  • v0.0.6(Feb 20, 2022)

    Added

    • Adds a command line interface for running a sequential search given a python script <script>.py containing a function main(config), a default configuration file <base>.yaml & a search configuration <search>.yaml. The main function should return a single scalar performance score. You can then start the search via:

      mle-search <script>.py --base_config <base>.yaml --search_config <search>.yaml --num_iters <search_iters>
      

      Or short via:

      mle-search <script>.py -base <base>.yaml -search <search>.yaml -iters <search_iters>
      
    • Adds doc-strings to all functionalities.

    Changed

    • Make it possible to optimize parameters in nested dictionaries. Added helpers flatten_config and unflatten_config. For shaping 'sub1/sub2/vname' <-> {sub1: {sub2: {vname: v}}}
    • Make start-up message also print fixed parameter settings.
    • Cleaned up decorator with the help of Strategies wrapper.
    Source code(tar.gz)
    Source code(zip)
  • v0.0.5(Jan 5, 2022)

    Added

    • Adds possibility to store and reload entire strategies as pkl file (as asked for in issue #2).
    • Adds improvement method indicating if score is better than best stored one
    • Adds save option for best plot
    • Adds args, kwargs into decorator
    • Adds synchronous Successive Halving (SuccessiveHalvingSearch - issue #3)
    • Adds synchronous HyperBand (HyperbandSearch - issue #3)
    • Adds synchronous PBT (PBTSearch - issue #4)
    • Adds option to save log in tell method
    • Adds small torch mlp example for SH/Hyperband/PBT w. logging/scheduler
    • Adds print welcome/update message for strategy specific info

    Changed

    • Major internal restructuring:
      • clean_data: Get rid of extra data provided in configuration file
      • tell_search: Update model of search strategy (e.g. SMBO/Nevergrad)
      • log_search: Add search specific log data to evaluation log
      • update_search: Refine search space/change active strategy etc.
    • Also allow to store checkpoint of trained models in tell method.
    • Fix logging message when log is stored
    • Make json serializer more robust for numpy data types
    • Robust type checking with isinstance(self.log[0]["objective"], (float, int, np.integer, np.float))
    • Update NB to include mle-scheduler example
    • Make PBT explore robust for integer/categorical valued hyperparams
    • Calculate total batches & their sizes for hyperband
    Source code(tar.gz)
    Source code(zip)
  • v0.0.3(Oct 24, 2021)

    • Fixes CoordinateSearch active grid search dimension updating. We have to account for the fact that previous coordinates are not evaluated again after switching the active variable.
    • Generalizes NevergradSearch to wrap around all search strategies.
    • Adds rich logging to all console print statements.
    • Updates documentation and adds text to getting_started.ipynb.
    Source code(tar.gz)
    Source code(zip)
  • v0.0.2(Oct 20, 2021)

    • Fixes import bug when using PyPi installation.
    • Enhances documentation and test coverage.
    • Adds search space refinement for nevergrad and smbo search strategies via refine_after and refine_top_k:
    strategy = SMBOSearch(
            real={"lrate": {"begin": 0.1, "end": 0.5, "prior": "uniform"}},
            integer={"batch_size": {"begin": 1, "end": 5, "prior": "uniform"}},
            categorical={"arch": ["mlp", "cnn"]},
            search_config={
                "base_estimator": "GP",
                "acq_function": "gp_hedge",
                "n_initial_points": 5,
                "refine_after": 5,
                "refine_top_k": 2,
            },
            seed_id=42,
            verbose=True
        )
    
    • Adds additional strategy boolean option maximize_objective to maximize instead of performing default black-box minimization.
    Source code(tar.gz)
    Source code(zip)
  • v0.0.1(Oct 16, 2021)

    Base API implementation:

    from mle_hyperopt import RandomSearch
    
    # Instantiate random search class
    strategy = RandomSearch(real={"lrate": {"begin": 0.1,
                                            "end": 0.5,
                                            "prior": "log-uniform"}},
                            integer={"batch_size": {"begin": 32,
                                                    "end": 128,
                                                    "prior": "uniform"}},
                            categorical={"arch": ["mlp", "cnn"]})
    
    # Simple ask - eval - tell API
    configs = strategy.ask(5)
    values = [train_network(**c) for c in configs]
    strategy.tell(configs, values)
    
    Source code(tar.gz)
    Source code(zip)
A 35mm camera, based on the Canonet G-III QL17 rangefinder, simulated in Python.

c is for Camera A 35mm camera, based on the Canonet G-III QL17 rangefinder, simulated in Python. The purpose of this project is to explore and underst

Daniele Procida 146 Sep 26, 2022
IPATool-py: download ipa easily

IPATool-py Python version of IPATool! Installation pip3 install -r requirements.txt Usage Quickstart: download app with specific bundleId into DIR: p

159 Dec 30, 2022
NeWT: Natural World Tasks

NeWT: Natural World Tasks This repository contains resources for working with the NeWT dataset. โ— At this time the binary tasks are not publicly avail

Visipedia 26 Oct 18, 2022
This is the pytorch code for the paper Curious Representation Learning for Embodied Intelligence.

Curious Representation Learning for Embodied Intelligence This is the pytorch code for the paper Curious Representation Learning for Embodied Intellig

19 Oct 19, 2022
Official implementation for the paper "Attentive Prototypes for Source-free Unsupervised Domain Adaptive 3D Object Detection"

Attentive Prototypes for Source-free Unsupervised Domain Adaptive 3D Object Detection PyTorch code release of the paper "Attentive Prototypes for Sour

Deepti Hegde 23 Oct 17, 2022
The PyTorch implementation for paper "Neural Texture Extraction and Distribution for Controllable Person Image Synthesis" (CVPR2022 Oral)

ArXiv | Get Start Neural-Texture-Extraction-Distribution The PyTorch implementation for our paper "Neural Texture Extraction and Distribution for Cont

Ren Yurui 111 Dec 10, 2022
Implementation of the paper NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting.

Non-AR Spatial-Temporal Transformer Introduction Implementation of the paper NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series For

Chen Kai 66 Nov 28, 2022
A small demonstration of using WebDataset with ImageNet and PyTorch Lightning

A small demonstration of using WebDataset with ImageNet and PyTorch Lightning

Tom 50 Dec 16, 2022
This repo is a C++ version of yolov5_deepsort_tensorrt. Packing all C++ programs into .so files, using Python script to call C++ programs further.

yolov5_deepsort_tensorrt_cpp Introduction This repo is a C++ version of yolov5_deepsort_tensorrt. And packing all C++ programs into .so files, using P

41 Dec 27, 2022
Source code for CAST - Crisis Domain Adaptation Using Sequence-to-sequence Transformers (Accepted to ISCRAM 2021, CorePaper).

Source code for CAST: Crisis Domain Adaptation UsingSequence-to-sequenceTransformers (Paper, BibTeX, Accepted to ISCRAM 2021, CorePaper) Quick start D

Congcong Wang 0 Jul 14, 2021
Code for Universal Semi-Supervised Semantic Segmentation models paper accepted in ICCV 2019

USSS_ICCV19 Code for Universal Semi Supervised Semantic Segmentation accepted to ICCV 2019. Full Paper available at https://arxiv.org/abs/1811.10323.

Tarun K 68 Nov 24, 2022
CLUES: Few-Shot Learning Evaluation in Natural Language Understanding

CLUES: Few-Shot Learning Evaluation in Natural Language Understanding This repo contains the data and source code for baseline models in the NeurIPS 2

Microsoft 29 Dec 29, 2022
Fake News Detection Using Machine Learning Methods

Fake-News-Detection-Using-Machine-Learning-Methods Fake news is always a real and dangerous issue. However, with the presence and abundance of various

Achraf Safsafi 1 Jan 11, 2022
Lightweight library to build and train neural networks in Theano

Lasagne Lasagne is a lightweight library to build and train neural networks in Theano. Its main features are: Supports feed-forward networks such as C

Lasagne 3.8k Dec 29, 2022
Official git repo for the CHIRP project

CHIRP Project This is the official git repository for the CHIRP project. Pull requests are accepted here, but for the moment, the main repository is s

Dan Smith 77 Jan 08, 2023
PyTorch implementation of Weak-shot Fine-grained Classification via Similarity Transfer

SimTrans-Weak-Shot-Classification This repository contains the official PyTorch implementation of the following paper: Weak-shot Fine-grained Classifi

BCMI 60 Dec 02, 2022
COCO Style Dataset Generator GUI

A simple GUI-based COCO-style JSON Polygon masks' annotation tool to facilitate quick and efficient crowd-sourced generation of annotation masks and bounding boxes. Optionally, one could choose to us

Hans Krupakar 142 Dec 09, 2022
Official codes for the paper "Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech"

ResDAVEnet-VQ Official PyTorch implementation of Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech What is in this repo? M

Wei-Ning Hsu 21 Aug 23, 2022
Lightweight stereo matching network based on MobileNetV1 and MobileNetV2

MobileStereoNet: Towards Lightweight Deep Networks for Stereo Matching

Cognitive Systems Research Group 139 Nov 30, 2022
Dogs classification with Deep Metric Learning using some popular losses

Tsinghua Dogs classification with Deep Metric Learning 1. Introduction Tsinghua Dogs dataset Tsinghua Dogs is a fine-grained classification dataset fo

QuocThangNguyen 45 Nov 09, 2022