Adam with minor modifications which give significant improvement

Related tags

MiscellaneousBAdam
Overview

BAdam

Modification of Adam [1] optimizer with increased stability and better performance. Tricks used:

  1. Decoupled weight decay as in AdamW [2]. Such decoupling allows easier tuning of weight decay and learning rate. This implementation follows PyTorch and multiplies weight decay by learning rate, allowing simultaneous scheduling. Due to this typical values passed to optimizer should be much higher that with SGD, if you use 1e-4 with SGD, good start would be to use 1e-2 with BAdam.
  2. Epsilon is inside sqrt to avoid NaN in mixed precision. Default value is much larger than in Adam to reduce 'adaptivity' it leads to better and wider optimums [3]. Large epsilon also works better than amsgrad version of Adam [5]
  3. exp_avg_sq inits with large value, rather than with zeros. This removes the need for lr warmup and does the same thing as all the tricks from RAdam [4], while being much simpler.
  4. Removed bias correction. It's not needed if exp_avg_sq is correctly initialized

Practical Tips

Default values for this optimizer were tuned on Imagenet and work as good baseline for other computer vision tasks. Try them as is, before further tuning.

Installation

pip install git+https://github.com/bonlime/[email protected]

Reference:

[1] Adam: A Method for Stochastic Optimization
[2] Decoupled Weight Decay Regularization
[3] On the Convergence of Adam and Beyond
[4] On the Variance of the Adaptive Learning Rate and Beyond [5] Adaptive Methods for Non-convex Optimization

Owner
Emil Zakirov. MIPT & Skoltech. Computer Vision Engineer.
Translation patch for Hololive ERROR

Translation patch for Hololive ERROR How do I install the patch? Grab the Translation.zip file for the latest version from the releases page, and unzi

18 Jul 20, 2022
A nonebot2 plugin, send news information in a picture form.

A nonebot2 plugin, send news information in a picture form.

幼稚园园长 7 Nov 18, 2022
Код файнтюнинга оригинального CLIP на русский язык

О чем репозиторий В этом репозитории представлен способ файтюнить оригинальный CLIP на новый язык Почему модель не видит женщину и откуда на картинке

Valentina Biryukova 7 Feb 06, 2022
An universal linux port of deezer, supporting both Flatpak and AppImage

Deezer for linux This repo is an UNOFFICIAL linux port of the official windows-only Deezer app. Being based on the windows app, it allows downloading

Aurélien Hamy 154 Jan 06, 2023
Repo to demo translating colab/jupyter notebook to streamlit webapp

Repo to demo translating colab/jupyter notebook to streamlit webapp

Marisa Smith 2 Feb 02, 2022
Provide error messages for Python exceptions, even if the original message is empty

errortext is a Python package to provide error messages for Python exceptions, even if the original message is empty.

Thomas Aglassinger 0 Dec 07, 2021
Python programs, usually short, of considerable difficulty, to perfect particular skills.

Peter Norvig MIT License 2015-2020 pytudes "An étude (a French word meaning study) is an instrumental musical composition, usually short, of considera

Peter Norvig 19.9k Dec 27, 2022
my own python useful functions

PythonToolKit Motivation This Repo should help save time for data scientists' daily work regarding the Time Series regression task by providing functi

Kai 2 Oct 01, 2022
How to use Microsoft Bing to search for leaks?

Installation In order to install the project, you need install its dependencies: $ pip3 install -r requirements.txt Add your Bing API key to bingKey.t

Ernestas Kardzys 2 Sep 21, 2022
Excel cell checker with python

excel-cell-checker Description This tool checks a given .xlsx file has the struc

Paul Aumann 1 Jan 04, 2022
This repository contains code for building education startup.

Learning Management System Overview It's the code for EssayBrain, a tool for teacher that automatically grades and validates essays. In order to valid

Shyam Das Shrestha 1 Nov 21, 2021
A tool for RaceRoom Racing Experience which shows you launch data

R3E Launch Tool A tool for RaceRoom Racing Experience which shows you launch data. Usage Run the tool, change the Stop Speed to whatever you want, and

Yuval Rosen 2 Feb 01, 2022
A TODO-list tool written in Python

PyTD A TODO-list tool written in Python. Its goal is to provide a stable posibility to get a good view over all your TODOs motivate you to actually fi

1 Feb 12, 2022
Proyecto desarrollado para el programa #FutureDevelopers, tabla periódica interactiva.

Tabla_Periodica Proyecto desarrollado para el programa #FutureDevelopers, tabla periódica interactiva. Descripcion primer entregable: Tabla periodica

1 Dec 04, 2021
Security-related flags and options for C compilers

Getting the maximum of your C compiler, for security

135 Nov 11, 2022
This repository containing cross-section cut and fill calculations using Python programming language.

cross-section This repository is containing cut and fill calculations for cross-section using Python programming language. This codes is made to calcu

3 Jun 15, 2022
Data Orchestration Platform

Table of contents What is DOP Design Concept A Typical DOP Orchestration Flow Prerequisites - Run in Docker For DOP Native Features For DBT Instructio

Datatonic 61 Mar 04, 2022
Back-end API for the reternal framework

RE:TERNAL RE:TERNAL is a centralised purple team simulation platform. Reternal uses agents installed on a simulation network to execute various known

Joey Dreijer 7 Apr 15, 2022
A brainfuck-based game oriented language written in python.

GF.py STILL WIP Gamefuck.py is a programming language based off brainfuck. It is oriented towards game development, and as such has many commands spec

Xenon 1 Feb 23, 2022
A streamlit app for exploring image search results from HuggingPics

title emoji colorFrom colorTo sdk app_file pinned huggingpics-explorer 🤗 blue red streamlit app.py false huggingpics-explorer A streamlit app for exp

Nathan Raw 4 Sep 10, 2022