PyTorch implementation of the paper Dynamic Token Normalization Improves Vision Transfromers.

Last update: Oct 09, 2022

Related tags

Overview

Dynamic Token Normalization Improves Vision Transformers

This is the PyTorch implementation of the paper Dynamic Token Normalization Improves Vision Transfromers. Codea and Models will be available soon.

Dynamic Token Normalization

We design a novel normalization method, termed Dynamic Token Normalization (DTN), which inherits the advantages from LayerNorm and InstanceNorm. DTN can be seamlessly plugged into various transformer models, consistenly improving the performance.

Comparisons of top-1 accuracies on the validation set of ImageNet, by using ViT trained with LN and DTN.

Model	Top-1	Top-5
ViT-T*-LN	72.3	91.4
ViT-T*-DTN	73.2	91.7
ViT-S*-LN	80.6	95.2
ViT-S*-DTN	81.7	95.8
ViT-B*-LN	81.7	95.8
ViT-B*-DTN	82.5	96.1

Getting Started

Install PyTorch

Clone the repo:

git clone https://github.com/dtn-anonymous/DTN.git

Requirements

Install CUDA==10.1 with cudnn7 following the official installation instructions
Install PyTorch==1.7.1 and torchvision==0.8.2 with CUDA==10.1:

conda install pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.1 -c pytorch

Install timm==0.3.2:

pip install timm==0.3.2

Data Preparation

Download the ImageNet dataset which should contain train and val directionary and the txt file for correspondings between images and labels.

Training a model from scratch

An example to train our DTN is given in DTN/scripts/train.sh. To train ViT-S* with our DTN,

cd DTN/scripts   
sh train.sh layer vit_norm_s_star configs/ViT/vit.yaml

Number of GPUs and configuration file to use can be modified in train.sh

PyTorch implementation of the paper Dynamic Token Normalization Improves Vision Transfromers.

Related tags

Overview

Dynamic Token Normalization Improves Vision Transformers

Dynamic Token Normalization

Getting Started

Requirements

Data Preparation

Training a model from scratch

Owner

Wenqi Shao

CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching（CVPR2021）

Benchmark for Answering Existential First Order Queries with Single Free Variable

Towards Understanding Quality Challenges of the Federated Learning: A First Look from the Lens of Robustness

Code for the ICML 2021 paper "Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation", Haoxiang Wang, Han Zhao, Bo Li.

A PyTorch implementation of "ANEMONE: Graph Anomaly Detection with Multi-Scale Contrastive Learning", CIKM-21

Performance Analysis of Multi-user NOMA Wireless-Powered mMTC Networks: A Stochastic Geometry Approach

Py-faster-rcnn - Faster R-CNN (Python implementation)

A Peer-to-peer Platform for Secure, Privacy-preserving, Decentralized Data Science

⚾🤖⚾ Automatic baseball pitching overlay in realtime

The Pytorch implementation for "Video-Text Pre-training with Learned Regions"

Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth

Code, environments, and scripts for the paper: "How Private Is Your RL Policy? An Inverse RL Based Analysis Framework"

Soft actor-critic is a deep reinforcement learning framework for training maximum entropy policies in continuous domains.

An Unbiased Learning To Rank Algorithms (ULTRA) toolbox

Part-Aware Data Augmentation for 3D Object Detection in Point Cloud

Rainbow is all you need! A step-by-step tutorial from DQN to Rainbow

Pytorch implementation of DeepMind's differentiable neural computer paper.

Adjusting for Autocorrelated Errors in Neural Networks for Time Series

Repository containing detailed experiments related to the paper "Memotion Analysis through the Lens of Joint Embedding".

Codeflare - Scale complex AI/ML pipelines anywhere