Adam with minor modifications which give significant improvement

Related tags

MiscellaneousBAdam
Overview

BAdam

Modification of Adam [1] optimizer with increased stability and better performance. Tricks used:

  1. Decoupled weight decay as in AdamW [2]. Such decoupling allows easier tuning of weight decay and learning rate. This implementation follows PyTorch and multiplies weight decay by learning rate, allowing simultaneous scheduling. Due to this typical values passed to optimizer should be much higher that with SGD, if you use 1e-4 with SGD, good start would be to use 1e-2 with BAdam.
  2. Epsilon is inside sqrt to avoid NaN in mixed precision. Default value is much larger than in Adam to reduce 'adaptivity' it leads to better and wider optimums [3]. Large epsilon also works better than amsgrad version of Adam [5]
  3. exp_avg_sq inits with large value, rather than with zeros. This removes the need for lr warmup and does the same thing as all the tricks from RAdam [4], while being much simpler.
  4. Removed bias correction. It's not needed if exp_avg_sq is correctly initialized

Practical Tips

Default values for this optimizer were tuned on Imagenet and work as good baseline for other computer vision tasks. Try them as is, before further tuning.

Installation

pip install git+https://github.com/bonlime/[email protected]

Reference:

[1] Adam: A Method for Stochastic Optimization
[2] Decoupled Weight Decay Regularization
[3] On the Convergence of Adam and Beyond
[4] On the Variance of the Adaptive Learning Rate and Beyond [5] Adaptive Methods for Non-convex Optimization

Owner
Emil Zakirov. MIPT & Skoltech. Computer Vision Engineer.
HungryBall to prosta gra, w której gracz wciela się w piłkę.

README POLSKI Opis gry HungryBall to prosta gra, w której gracz wciela się w piłkę. Sterowanie odbywa się za pomocą przycisków w, a, s i d lub opcjona

Karol 1 Nov 24, 2021
Python Control Systems Library

The Python Control Systems Library is a Python module that implements basic operations for analysis and design of feedback control systems.

Control Systems Library for Python 1.3k Jan 06, 2023
Easy Alias's for bash

easy-alias Easy Alias's for bash Setup Your system needs to have 'echo' which every 21st century computer has You dont need any python requirments but

Hashm 2 Jan 18, 2022
A web project to control the daily life budget planing

Budget Planning - API In this repo there's only the API and Back-End of the this project. Install and run the project # install virtualenv --python=py

Leonardo Da Vinci 1 Oct 24, 2021
Курс про техническое совершенство для нетехнарей

Technical Excellence 101 Курс про техническое совершенство для нетехнарей. Этот курс представлят из себя серию воркшопов, при помощи которых можно объ

Anton Bevzuk 11 Nov 13, 2022
Final project in KAIST AI class

mmodal_mixer MLP-Mixer based Multi-modal image-text retrieval Image: Original image is cropped with 16 x 16 patch size without overlap. Then, it is re

SuperSuperMoon 5 May 30, 2022
The dynamic code loading framework used in LocalStack

localstack-plugin-loader localstack-plugin-loader is the dynamic code loading framework used in LocalStack. Install pip install localstack-plugin-load

LocalStack 5 Oct 09, 2022
Find all social media accounts with a username!

Aliens_eye FIND ALL SOCIAL MEDIA ACCOUNTS WITH A USERNAME! OSINT To install: Open terminal and type: git clone https://github.com/BLINKING-IDIOT/Alien

Aaron Thomas 84 Dec 28, 2022
ASCII-Wordle - A port of the game Wordle to terminal emulators/CMD

ASCII-Wordle A 'port' of Wordle to text-based interfaces A near-feature complete

32 Jun 11, 2022
PhD document for navlab

PhD_document_for_navlab The project contains the relative software documents which I developped or used during my PhD period. It includes: FLVIS. A st

ZOU YAJING 9 Feb 21, 2022
A system for assigning and grading notebooks

nbgrader Linux: Windows: Forum: Coverage: Cite: A system for assigning and grading Jupyter notebooks. Documentation can be found on Read the Docs. Hig

Project Jupyter 1.2k Dec 26, 2022
MuMMI Core is the underlying infrastructure and generalizable component of the MuMMI framework

MuMMI Core is the underlying infrastructure and generalizable component of the MuMMI framework, which facilitates the coordination of massively parallel multiscale simulations.

4 Aug 17, 2022
Blender addon for executing the operator in response to the received OSC message.

I/F Joiner 受信したOSCメッセージに応じてオペレータ(bpy.ops)を実行するアドオンです. OSC通信に対応したコントローラやアプリをインストールしたスマートフォンを使用してBlenderを操作することが可能になります. 同時開発しているAndroidコントローラ化アプリMocopa

simasimataiyo 6 Oct 02, 2022
A PG3D API Made with Python

PG3D Python API A Pixel Gun 3D Python API (Public Ver) Features Count: 29 How To Use? import api as pbn Examples pbn.isBanned(192819483) - True pbn.f

Karim 2 Mar 24, 2022
A collection of Workflows samples for various use cases

Workflows Samples Workflows allow you to orchestrate and automate Google Cloud and HTTP-based API services with serverless workflows.

Google Cloud Platform 76 Jan 07, 2023
Job Guy Backend

جاب‌گای چیست؟ اونجا وضعیت چطوریه؟ یه سوال به همین کلیت و ابهام معمولا وقتی برای یه شرکت رزومه می‌فرستیم این سوال کلی و بزرگ برای همه پیش میاد.اونجا وض

Jobguy.work 217 Dec 25, 2022
A data driven app for bicycle hiring in London(UK)

bicycle_hiring_app_deployed A data driven app for bicycle hiring in London(UK). It predicts expected number of bicycle hire in London. It asks users t

Rajarshi Roy Raju 1 Dec 10, 2021
This script can be used to get unlimited Gb for WARP.

Warp-Unlimited-GB This script can be used to get unlimited Gb for WARP. How to use Change the value of the 'referrer' to warp id of yours You can down

Anix Sam Saji 1 Feb 14, 2022
A similarity measurer on two programming assignments on Online Judge.

A similarity measurer on two programming assignments on Online Judge. Algorithm implementation details are at here. Install Recommend OS: Ubuntu 20.04

StardustDL 6 May 21, 2022