Exploring whether attention is necessary for vision transformers

Last update: Jan 07, 2023

Overview

Do You Even Need Attention?
A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet

Paper/Report

TL;DR

We replace the attention layer in a vision transformer with a feed-forward layer and find that it still works quite well on ImageNet.

Abstract

The strong performance of vision transformers on image classification and other vision tasks is often attributed to the design of their multi-head attention layers. However, the extent to which attention is responsible for this strong performance remains unclear. In this short report, we ask: is the attention layer even necessary? Specifically, we replace the attention layer in a vision transformer with a feed-forward layer applied over the patch dimension. The resulting architecture is simply a series of feed-forward layers applied over the patch and feature dimensions in an alternating fashion. In experiments on ImageNet, this architecture performs surprisingly well: a ViT/DeiT-base-sized model obtains 74.9% top-1 accuracy, compared to 77.9% and 79.9% for ViT and DeiT respectively. These results indicate that aspects of vision transformers other than attention, such as the patch embedding, may be more responsible for their strong performance than previously thought. We hope these results prompt the community to spend more time trying to understand why our current models are as effective as they are.

Note

This is concurrent research with MLP-Mixer from Google Research. The ideas are exacty the same, with the one difference being that they use (a lot) more compute.

Pretrained models and logs

Here is a Weights and Biases report with the expected training trajectory: W&B

name	[email protected]	#params	url
FF-tiny	61.4	7.7M	model
FF-base	74.9	62M	model
FF-large	71.4	206M	-

Note: I haven't uploaded the FF-Large model because (1) it's over GitHub's file storage limit, and (2) I don't see why anyone would want it, given that it performs worse than the base model. That being said, if you want it, reach out to me and I'll send it to you.

How to train

The model definition in vision_transformer_linear.py is designed to be run with the repo from DeiT, which is itself based on the wonderful timm package.

Steps:

Clone the DeiT repo and move the file into it

git clone https://github.com/facebookresearch/deit
mv vision_transformer_linear.py deit
cd deit

Add a line to import vision_transformer_linear in main.py. For example, add the following after the import statements (around line 27):

+ import vision_transformer_linear

Train:

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m torch.distributed.launch \
--nproc_per_node=8 \
--master_port 10490 \
--use_env main.py \
--model linear_tiny \
--batch-size 128 \
--drop 0.1 \
--output_dir outputs/linear-tiny \
--data-path your/path/to/imagenet

Citation

If you build upon this idea, feel free to drop a citation (and also cite MLP-Mixer).

@article{melaskyriazi2021doyoueven,
  title={Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet},
  author={Luke Melas-Kyriazi},
  journal=arxiv,
  year=2021
}

Comments

On details about the experiments

In the experiment section of the report, you mention:

Such a comparison is not exactly fair because the feed-forward model uses stronger training augmentations.

I wonder what the augmentations are, for details about the augmentation seems missing from the paper.

opened by boredtylin 2
Positional Encoding Ablation

Hi Luke, thank you for sharing this amazing work.

In your arxiv document, I cannot find any mention of positional encoding, but I see that you use them in your code. Did you conduct any ablation study on the PE? i.e., how much does it affect the performance, with and without?

Thank you in advance.

opened by jjparkcv 0
Interaction between patches through a transpose may have a stronger role to play ?

Hi, I was going through your exp report. You have made a point that since you are able to get a good performance without using attention layer so good performance of ViT may be more to do with it's embedding layer than attention .

But I believe, It's also may be to do with how you have established an interaction between patches through a transpose very similar to what was done in MLP-Mixer .

Would love to know your thoughts on this ?

opened by rakshith291 0

The first dataset of composite images with rationality score indicating whether the object placement in a composite image is reasonable.

Object-Placement-Assessment-Dataset-OPA Object-Placement-Assessment (OPA) is to verify whether a composite image is plausible in terms of the object p

53 Nov 15, 2022

Face Mask Detection is a project to determine whether someone is wearing mask or not, using deep neural network.

face-mask-detection Face Mask Detection is a project to determine whether someone is wearing mask or not, using deep neural network. It contains 3 scr

13 Jan 18, 2022

This program was designed to detect whether someone is wearing a facemask through a live video stream.

This program was designed to detect whether someone is wearing a facemask through a live video stream. A custom lightweight CNN trained with TensorFlow on a public dataset provided by Kaggle is used to detect whether each face detected by the cv2 face detection dnn is wearing a mask

0 Apr 2, 2022

A system used to detect whether a person is wearing a medical mask or not.

Mask_Detection_System A system used to detect whether a person is wearing a medical mask or not. To open the program, please follow these steps: Make

0 Nov 17, 2022

Diabet Feature Engineering - Predict whether people have diabetes when their characteristics are specified

6 Jan 18, 2022

Diabetes-Feature-Engineering - A machine learning model that can predict whether people have diabetes when their characteristics are specified

Diabetes-Feature-Engineering Aim Developing a machine learning model that can pr

0 Feb 23, 2022

Multivariate Time Series Forecasting with efficient Transformers. Code for the paper "Long-Range Transformers for Dynamic Spatiotemporal Forecasting."

Spacetimeformer Multivariate Forecasting This repository contains the code for the paper, "Long-Range Transformers for Dynamic Spatiotemporal Forecast

440 Jan 2, 2023

Implementation of SE3-Transformers for Equivariant Self-Attention, in Pytorch.

SE3 Transformer - Pytorch Implementation of SE3-Transformers for Equivariant Self-Attention, in Pytorch. May be needed for replicating Alphafold2 resu

207 Dec 23, 2022

Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.

PyTorch Implementation of Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers 1 Using Colab Please notic

489 Jan 7, 2023

Exploring whether attention is necessary for vision transformers

Related tags

Overview

Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet

Paper/Report

TL;DR

Abstract

Note

Pretrained models and logs

How to train

Citation

You might also like...

The first dataset of composite images with rationality score indicating whether the object placement in a composite image is reasonable.

Face Mask Detection is a project to determine whether someone is wearing mask or not, using deep neural network.

This program was designed to detect whether someone is wearing a facemask through a live video stream.

A system used to detect whether a person is wearing a medical mask or not.

Diabet Feature Engineering - Predict whether people have diabetes when their characteristics are specified

Diabetes-Feature-Engineering - A machine learning model that can predict whether people have diabetes when their characteristics are specified

Multivariate Time Series Forecasting with efficient Transformers. Code for the paper "Long-Range Transformers for Dynamic Spatiotemporal Forecasting."

Implementation of SE3-Transformers for Equivariant Self-Attention, in Pytorch.

Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.

Comments

On details about the experiments

Positional Encoding Ablation

Interaction between patches through a transpose may have a stronger role to play ?

Releases(v0.0.1)

v0.0.1(May 6, 2021)

Owner

Luke Melas-Kyriazi

v objective diffusion inference code for JAX.

La source de mon module 'pyfade' disponible sur Pypi.

Kaggleship: Kaggle Notebooks

根据midi文件演奏“风物之诗琴”的脚本 "Windsong Lyre" auto play

PConv-Keras - Unofficial implementation of "Image Inpainting for Irregular Holes Using Partial Convolutions". Try at: www.fixmyphoto.ai

Custom TensorFlow2 implementations of forward and backward computation of soft-DTW algorithm in batch mode.

Re-implementation of the Noise Contrastive Estimation algorithm for pyTorch, following "Noise-contrastive estimation: A new estimation principle for unnormalized statistical models." (Gutmann and Hyvarinen, AISTATS 2010)

Trading environnement for RL agents, backtesting and training.

Tensorflow implementation of DeepLabv2

A copy of Ares that costs 30 fucking dollars.

This project provides an unsupervised framework for mining and tagging quality phrases on text corpora with pretrained language models (KDD'21).

Model-free Vehicle Tracking and State Estimation in Point Cloud Sequences

Transformer in Vision

PyTorch implementation HoroPCA: Hyperbolic Dimensionality Reduction via Horospherical Projections

Code for ICCV2021 paper PARE: Part Attention Regressor for 3D Human Body Estimation

WarpRNNT loss ported in Numba CPU/CUDA for Pytorch

SimBERT升级版（SimBERTv2）！

A implemetation of the LRCN in mxnet

Neural-net-from-scratch - A simple Neural Network from scratch in Python using the Pymathrix library

Complete system for facial identity system. Include one-shot model, database operation, features visualization, monitoring

Do You Even Need Attention?
A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet