Parameter-ensemble-differential-evolution - Shows how to do parameter ensembling using differential evolution.

Last update: May 04, 2022

Overview

Ensembling parameters with differential evolution

This repository shows how to ensemble parameters of two trained neural networks using differential evolution. The steps followed are as follows:

Train two networks (architecturally same) on the same dataset (CIFAR-10 used here) but from two different random initializations.
Ensemble their weights using the following formulae:
```
w_t = w_o * ema + (1 - ema) * w_p
```
w_o and w_p represents the learned of a neural network.
Randomly initialize a network (same architecture as above) and populate its parameters w_t using the above formulae.

ema is usually chosen by the developer in an empirical manner. This project uses differential evolution to find it.

Below are the top-1 accuracies (on CIFAR-10 test set) of two individually trained two models along with their ensembled variant:

Model one: 63.23%
Model two: 63.42%
Ensembled: 63.35%

With the more conventional average prediction ensembling, I was able to get to 64.92%. This is way better than what I got by ensembling the parameters. Nevertheless, the purpose of this project was to just try out an idea.

Reproducing the results

Ensure the requirements.txt is satisfied. Then train two models with ensuring your working directory is at the root of this project:

$ git clone https://github.com/sayakpaul/parameter-ensemble-differential-evolution
$ cd parameter-ensemble-differential-evolution
$ pip install -qr requirements.txt
$ for i in `seq 1 2`; python train.py; done

Then just follow the ensemble-parameters.ipynb notebook. You can also use the networks I trained. Instructions are available inside the notebook.

Parameter-ensemble-differential-evolution - Shows how to do parameter ensembling using differential evolution.

Related tags

Overview

Ensembling parameters with differential evolution

Reproducing the results

References

You might also like...

Neural Ensemble Search for Performant and Calibrated Predictions

An Ensemble of CNN (Python 3.5.1 Tensorflow 1.3 numpy 1.13)

zeus is a Python implementation of the Ensemble Slice Sampling method.

Pytorch implementation of SenFormer: Efficient Self-Ensemble Framework for Semantic Segmentation

Ensemble Knowledge Guided Sub-network Search and Fine-tuning for Filter Pruning

This Jupyter notebook shows one way to implement a simple first-order low-pass filter on sampled data in discrete time.

A fast Evolution Strategy implementation in Python

Code for the paper Task Agnostic Morphology Evolution.

Pytorch implementation of FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

Releases(v0.1.0)

v0.1.0(Jan 2, 2022)

Owner

Sayak Paul

LAMDA: Label Matching Deep Domain Adaptation

Massively parallel Monte Carlo diffusion MR simulator written in Python.

Code for the paper: Audio-Visual Scene Analysis with Self-Supervised Multisensory Features

A Planar RGB-D SLAM which utilizes Manhattan World structure to provide optimal camera pose trajectory while also providing a sparse reconstruction containing points, lines and planes, and a dense surfel-based reconstruction.

This is a repository for a semantic segmentation inference API using the OpenVINO toolkit

An Unpaired Sketch-to-Photo Translation Model

GitHub repository for "Improving Video Generation for Multi-functional Applications"

PyTorch implementation and pretrained models for XCiT models. See XCiT: Cross-Covariance Image Transformer

PyTorch implementation of "Efficient Neural Architecture Search via Parameters Sharing"

Code for Environment Dynamics Decomposition (ED2).

Pytorch0.4.1 codes for InsightFace

Github for the conference paper GLOD-Gaussian Likelihood OOD detector

Official and maintained implementation of the paper "OSS-Net: Memory Efficient High Resolution Semantic Segmentation of 3D Medical Data" [BMVC 2021].

Official PyTorch Implementation of Mask-aware IoU and maYOLACT Detector [BMVC2021]

Nest Protect integration for Home Assistant. This will allow you to integrate your smoke, heat, co and occupancy status real-time in HA.

A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)

Self-supervised Point Cloud Prediction Using 3D Spatio-temporal Convolutional Networks

Deep Hedging Demo - An Example of Using Machine Learning for Derivative Pricing.

LieTransformer: Equivariant Self-Attention for Lie Groups

Video lie detector using xgboost - A video lie detector using OpenFace and xgboost