Pytorch implementation of AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

Last update: Sep 16, 2022

Related tags

Deep Learning AngularGrad

Overview

AngularGrad Optimizer

This repository contains the oficial implementation for AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks in PyTorch.

AngularGrad reduces the zig-zag effect in the optimization trajectory. Fluctuations are significantly smoothed, tracing a more direct path towards the minimum of the cost function.

If you have questions or suggestions, please feel free to open an issue. Please cite as:

@article{roy2021angulargrad,
  title={AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks},
  author={S.K. Roy, M.E. Paoletti, J.M. Haut, S.R. Dubey, P. Kar, A. Plaza and B.B. Chaudhuri},
  journal={arXiv preprint arXiv:2105.10190},
  year={2021}
}

In PyTorch:

optimizer = cosangulargrad(params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0)
<or>
optimizer = tanangulargrad(params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0)
Example:
optimizer = cosangulargrad(net.parameters())

Experiments

Experiments in the paper:

Analitycal

cd analitycal/
python main.py

CIFAR-10/100

cd cifar/
python main.py --dataset <cifar10/cifar100> --model <r18/r34/.../vgg16/d121> --alg <adam/sgd/.../cosangulargrad/tanangulargrad> --lr <float>
Example:
python main.py --dataset cifar10 --model r50 --alg cosangulargrad --lr 1e-3

Mini-ImageNet:

cd mini-imagenet/
wget URL dataset
python main.py DATADIR --alg <adam/sgd/.../cosangulargrad/tanangulargrad> --lr <float>
Example:
python main.py ./split_mini/ --alg cosangulargrad --model r50 --lr 1e-3

Fine-Grained:

cd fine-grained/
wget URL datasets
python main.py DATADIR --dataset <dsetname> --alg <adam/sgd/.../cosangulargrad/tanangulargrad> --lr <float>
Example:
python main.py ./data/Car196/ --dataset cars --alg adam --lr 1e-3

Pytorch implementation of AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

Related tags

Overview

AngularGrad Optimizer

Experiments

Owner

mario

Self-training with Weak Supervision (NAACL 2021)

Code To Tune or Not To Tune? Zero-shot Models for Legal Case Entailment.

Parameter Efficient Deep Probabilistic Forecasting

Code release for the ICML 2021 paper "PixelTransformer: Sample Conditioned Signal Generation".

Using this codebase as a tool for my own research. Making some modifications to the original repo for my own purposes.

Pytorch tutorials for Neural Style transfert

Music Classification: Beyond Supervised Learning, Towards Real-world Applications

Official Pytorch implementation of the paper "Action-Conditioned 3D Human Motion Synthesis with Transformer VAE", ICCV 2021

Diverse Image Captioning with Context-Object Split Latent Spaces (NeurIPS 2020)

Bilinear attention networks for visual question answering

ObjectDrawer-ToolBox: a graphical image annotation tool to generate ground plane masks for a 3D object reconstruction system

Robot Servers and Server Manager software for robo-gym

sequitur is a library that lets you create and train an autoencoder for sequential data in just two lines of code

No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistency

Sdf sparse conv - Deep Learning on SDF for Classifying Brain Biomarkers

DeepFill v1/v2 with Contextual Attention and Gated Convolution, CVPR 2018, and ICCV 2019 Oral

Weakly Supervised Posture Mining with Reverse Cross-entropy for Fine-grained Classification

A PyTorch implementation of the paper "Semantic Image Synthesis via Adversarial Learning" in ICCV 2017

State of the Art Neural Networks for Generative Deep Learning

An Evaluation of Generative Adversarial Networks for Collaborative Filtering.