Revitalizing CNN Attention via Transformers in Self-Supervised Visual Representation Learning

Related tags

Deep LearningCARE
Overview

Revitalizing CNN Attention via Transformers in Self-Supervised Visual Representation Learning

This repository is the official implementation of CARE. Graph

Updates

  • (09/10/2021) Our paper is accepted by NeurIPS 2021.

Requirements

To install requirements:

conda create -n care python=3.6
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.1 -c pytorch
pip install tensorboard
pip install ipdb
pip install einops
pip install loguru
pip install pyarrow==3.0.0
pip install tqdm

๐Ÿ“‹ Pytorch>=1.6 is needed for runing the code.

Data Preparation

Prepare the ImageNet data in {data_path}/train.lmdb and {data_path}/val.lmdb

Relpace the original data path in care/data/dataset_lmdb (Line7 and Line40) with your new {data_path}.

๐Ÿ“‹ Note that we use the lmdb file to speed-up the data-processing procedure.

Training

Before training the ResNet-50 (100 epoch) in the paper, run this command first to add your PYTHONPATH:

export PYTHONPATH=$PYTHONPATH:{your_code_path}/care/
export PYTHONPATH=$PYTHONPATH:{your_code_path}/care/care/

Then run the training code via:

bash run_train.sh      #(The training script is used for trianing CARE with 8 gpus)
bash single_gpu_train.sh    #(We also provide the script for trainig CARE with only one gpu)

๐Ÿ“‹ The training script is used to do unsupervised pre-training of a ResNet-50 model on ImageNet in an 8-gpu machine

  1. using -b to specify batch_size, e.g., -b 128
  2. using -d to specify gpu_id for training, e.g., -d 0-7
  3. using --log_path to specify the main folder for saving experimental results.
  4. using --experiment-name to specify the folder for saving training outputs.

The code base also supports for training other backbones (e.g., ResNet101 and ResNet152) with different training schedules (e.g., 200, 400 and 800 epochs).

Evaluation

Before start the evaluation, run this command first to add your PYTHONPATH:

export PYTHONPATH=$PYTHONPATH:{your_code_path}/care/
export PYTHONPATH=$PYTHONPATH:{your_code_path}/care/care/

Then, to evaluate the pre-trained model (e.g., ResNet50-100epoch) on ImageNet, run:

bash run_val.sh      #(The training script is used for evaluating CARE with 8 gpus)
bash debug_val.sh    #(We also provide the script for evaluating CARE with only one gpu)

๐Ÿ“‹ The training script is used to do the supervised linear evaluation of a ResNet-50 model on ImageNet in an 8-gpu machine

  1. using -b to specify batch_size, e.g., -b 128
  2. using -d to specify gpu_id for training, e.g., -d 0-7
  3. Modifying --log_path according to your own config.
  4. Modifying --experiment-name according to your own config.

Pre-trained Models

We here provide some pre-trained models in the [shared folder]:

Here are some examples.

  • [ResNet-50 100epoch] trained on ImageNet using ResNet-50 with 100 epochs.
  • [ResNet-50 200epoch] trained on ImageNet using ResNet-50 with 200 epochs.
  • [ResNet-50 400epoch] trained on ImageNet using ResNet-50 with 400 epochs.

More models are provided in the following model zoo part.

๐Ÿ“‹ We will provide more pretrained models in the future.

Model Zoo

Our model achieves the following performance on :

Self-supervised learning on image classifications.

Method Backbone epoch Top-1 Top-5 pretrained model linear evaluation model
CARE ResNet50 100 72.02% 90.02% [pretrained] (wip) [linear_model] (wip)
CARE ResNet50 200 73.78% 91.50% [pretrained] (wip) [linear_model] (wip)
CARE ResNet50 400 74.68% 91.97% [pretrained] (wip) [linear_model] (wip)
CARE ResNet50 800 75.56% 92.32% [pretrained] (wip) [linear_model] (wip)
CARE ResNet50(2x) 100 73.51% 91.66% [pretrained] (wip) [linear_model] (wip)
CARE ResNet50(2x) 200 75.00% 92.22% [pretrained] (wip) [linear_model] (wip)
CARE ResNet50(2x) 400 76.48% 92.99% [pretrained] (wip) [linear_model] (wip)
CARE ResNet50(2x) 800 77.04% 93.22% [pretrained] (wip) [linear_model] (wip)
CARE ResNet101 100 73.54% 91.63% [pretrained] (wip) [linear_model] (wip)
CARE ResNet101 200 75.89% 92.70% [pretrained] (wip) [linear_model] (wip)
CARE ResNet101 400 76.85% 93.31% [pretrained] (wip) [linear_model] (wip)
CARE ResNet101 800 77.23% 93.52% [pretrained] (wip) [linear_model] (wip)
CARE ResNet152 100 74.59% 92.09% [pretrained] (wip) [linear_model] (wip)
CARE ResNet152 200 76.58% 93.63% [pretrained] (wip) [linear_model] (wip)
CARE ResNet152 400 77.40% 93.63% [pretrained] (wip) [linear_model] (wip)
CARE ResNet152 800 78.11% 93.81% [pretrained] (wip) [linear_model] (wip)

Transfer learning to object detection and semantic segmentation.

COCO det

Method Backbone epoch AP_bb AP_50 AP_75 pretrained model det/seg model
CARE ResNet50 200 39.4 59.2 42.6 [pretrained] (wip) [model] (wip)
CARE ResNet50 400 39.6 59.4 42.9 [pretrained] (wip) [model] (wip)
CARE ResNet50-FPN 200 39.5 60.2 43.1 [pretrained] (wip) [model] (wip)
CARE ResNet50-FPN 400 39.8 60.5 43.5 [pretrained] (wip) [model] (wip)

COCO instance seg

Method Backbone epoch AP_mk AP_50 AP_75 pretrained model det/seg model
CARE ResNet50 200 34.6 56.1 36.8 [pretrained] (wip) [model] (wip)
CARE ResNet50 400 34.7 56.1 36.9 [pretrained] (wip) [model] (wip)
CARE ResNet50-FPN 200 35.9 57.2 38.5 [pretrained] (wip) [model] (wip)
CARE ResNet50-FPN 400 36.2 57.4 38.8 [pretrained] (wip) [model] (wip)

VOC07+12 det

Method Backbone epoch AP_bb AP_50 AP_75 pretrained model det/seg model
CARE ResNet50 200 57.7 83.0 64.5 [pretrained] (wip) [model] (wip)
CARE ResNet50 400 57.9 83.0 64.7 [pretrained] (wip) [model] (wip)

๐Ÿ“‹ More results are provided in the paper.

Contributing

๐Ÿ“‹ WIP

Owner
ChongjianGE
๐ŸŽฏ PhD in Computer Vision โ˜‘๏ธ MSc & BEng in Electrical Engineering
ChongjianGE
Code For TDEER: An Efficient Translating Decoding Schema for Joint Extraction of Entities and Relations (EMNLP2021)

TDEER (WIP) Code For TDEER: An Efficient Translating Decoding Schema for Joint Extraction of Entities and Relations (EMNLP2021) Overview TDEER is an e

Alipay 6 Dec 17, 2022
A faster pytorch implementation of faster r-cnn

A Faster Pytorch Implementation of Faster R-CNN Write at the beginning [05/29/2020] This repo was initaited about two years ago, developed as the firs

Jianwei Yang 7.1k Jan 01, 2023
[NeurIPS '21] Adversarial Attacks on Graph Classification via Bayesian Optimisation (GRABNEL)

Adversarial Attacks on Graph Classification via Bayesian Optimisation @ NeurIPS 2021 This repository contains the official implementation of GRABNEL,

Xingchen Wan 12 Dec 23, 2022
Pytorch implementation of face attention network

Face Attention Network Pytorch implementation of face attention network as described in Face Attention Network: An Effective Face Detector for the Occ

Hooks 312 Dec 09, 2022
A collection of random and hastily hacked together scripts for investigating EU-DCC

A collection of random and hastily hacked together scripts for investigating EU-DCC

Ryan Barrett 8 Mar 01, 2022
The official repo of the CVPR 2021 paper Group Collaborative Learning for Co-Salient Object Detection .

GCoNet The official repo of the CVPR 2021 paper Group Collaborative Learning for Co-Salient Object Detection . Trained model Download final_gconet.pth

Qi Fan 46 Nov 17, 2022
Auto-updating data to assist in investment to NEPSE

Symbol Ratios Summary Sector LTP Undervalued Bonus % MEGA Strong Commercial Banks 368 5 10 JBBL Strong Development Banks 568 5 10 SIFC Strong Finance

Amit Chaudhary 16 Nov 01, 2022
DeepFaceLab fork which provides IPython Notebook to use DFL with Google Colab

DFL-Colab โ€” DeepFaceLab fork for Google Colab This project provides you IPython Notebook to use DeepFaceLab with Google Colaboratory. You can create y

779 Jan 05, 2023
EquiBind: Geometric Deep Learning for Drug Binding Structure Prediction

EquiBind: geometric deep learning for fast predictions of the 3D structure in which a small molecule binds to a protein

Hannes Stรคrk 355 Jan 03, 2023
Code for Environment Dynamics Decomposition (ED2).

ED2 Code for Environment Dynamics Decomposition (ED2). Installation Follow the installation in MBPO and Dreamer. Usage First follow the SD2 method for

0 Aug 10, 2021
NAACL2021 - COIL Contextualized Lexical Retriever

COIL Repo for our NAACL paper, COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List. The code covers learning

Luyu Gao 108 Dec 31, 2022
StyleGAN2-ADA - Official PyTorch implementation

Abstract: Training generative adversarial networks (GAN) using too little data typically leads to discriminator overfitting, causing training to diverge. We propose an adaptive discriminator augmenta

NVIDIA Research Projects 3.2k Dec 30, 2022
Cl datasets - PyTorch image dataloaders and utility functions to load datasets for supervised continual learning

Continual learning datasets Introduction This repository contains PyTorch image

berjaoui 5 Aug 28, 2022
Library to enable Bayesian active learning in your research or labeling work.

Bayesian Active Learning (BaaL) BaaL is an active learning library developed at ElementAI. This repository contains techniques and reusable components

ElementAI 687 Dec 25, 2022
DeepVoxels is an object-specific, persistent 3D feature embedding.

DeepVoxels is an object-specific, persistent 3D feature embedding. It is found by globally optimizing over all available 2D observations of

Vincent Sitzmann 196 Dec 25, 2022
โ€œData Augmentation for Cross-Domain Named Entity Recognitionโ€ (EMNLP 2021)

Data Augmentation for Cross-Domain Named Entity Recognition Authors: Shuguang Chen, Gustavo Aguilar, Leonardo Neves and Thamar Solorio This repository

<a href=[email protected]"> 18 Sep 10, 2022
This repo provides function call to track multi-objects in videos

Custom Object Tracking Introduction This repo provides function call to track multi-objects in videos with a given trained object detection model and

Jeff Lo 51 Nov 22, 2022
mlpack: a scalable C++ machine learning library --

a fast, flexible machine learning library Home | Documentation | Doxygen | Community | Help | IRC Chat Download: current stable version (3.4.2) mlpack

mlpack 4.2k Jan 09, 2023
TensorFlow implementation of ENet

TensorFlow-ENet TensorFlow implementation of ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation. This model was tested on th

Kwotsin 255 Oct 17, 2022
GeneDisco is a benchmark suite for evaluating active learning algorithms for experimental design in drug discovery.

GeneDisco is a benchmark suite for evaluating active learning algorithms for experimental design in drug discovery.

22 Dec 12, 2022