Implementation for paper "Towards the Generalization of Contrastive Self-Supervised Learning"

Last update: Nov 30, 2022

Overview

Contrastive Self-Supervised Learning on CIFAR-10

Paper

"Towards the Generalization of Contrastive Self-Supervised Learning", Weiran Huang, Mingyang Yi and Xuyang Zhao, arXiv:2111.00743, 2021.

The most critical argument we made in our paper is that the quality of data augmentation exhibits great impact on the quality of contrastive-learned encoder. The data augmentation with sharper intra-class concentration enables the model to have better generalization on downstream tasks. We verify it through a variety of experiments in this repository.

Supported methods

SimCLR
Barlow Twins
MoCo
SimSiam

Installation

pip install -r requirement.txt

Dependencies

torch==1.4.0
torchvision==0.5.0
pytorch-lightning==1.3.8
lightly==1.0.8 (important!)

Evaluation

KNN evaluation protocol. Code from here.

Results

ResNet-18 trained by SimCLR and Barlow Twins over various data augmentation combinations.

Example: python main.py --model=twins --epoch=800 --batch=512 --round=3 --augs=abcde

(a)	(b)	(c)	(d)	(e)	SimCLR	Barlow Twins
✓	✓	✓	✓	✓	89.92 ± 0.05	83.93 ± 0.57
✓	✓	✓	✓	×	88.41 ± 0.11	83.37 ± 0.43
✓	✓	✓	×	×	83.62 ± 0.19	73.70 ± 0.99
✓	✓	×	×	×	62.91 ± 0.25	49.56 ± 0.11
✓	×	×	×	×	62.37 ± 0.09	48.54 ± 0.29

Augmentation operations include:

(a) random cropping with a scaling factor chosen in [0.08, 1.0];

(b) random Gaussian blur with a probability 0.5;

(d) color distortion with a probability of 0.8 and with strength of [0.4, 0.4, 0.4, 0.1];

(e) random horizontal flipping with a probability of 0.5.

ResNet18 trained by SimCLR and Barlow Twins over various color distortion strengths.

Example: python main.py --model=simclr --epoch=800 --batch=512 --round=3 --augs=color --strength=1

Color Distortion Strength	SimCLR	Barlow Twins
1/8	73.60 ± 0.11	61.13 ± 2.81
1/4	76.25 ± 0.16	68.30 ± 0.15
1/2	78.49 ± 0.09	72.76 ± 1.50
1	82.64 ± 0.57	78.79 ± 0.54

Acknowledgement

This code is based on:

Citation

If you find our work useful in your research, please consider citing:

@misc{huang2021generalization,
      title={Towards the Generalization of Contrastive Self-Supervised Learning}, 
      author={Weiran Huang and Mingyang Yi and Xuyang Zhao},
      year={2021},
      eprint={2111.00743},
      archivePrefix={arXiv}
}

You might also like...

Implementation of Nyström Self-attention, from the paper Nyströmformer

Nyström Attention Implementation of Nyström Self-attention, from the paper Nyströmformer. Yannic Kilcher video Install $ pip install nystrom-attention

95 Jan 2, 2023

Implementation of SETR model, Original paper: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.

SETR - Pytorch Since the original paper (Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.) has no official

112 Dec 16, 2022

Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

CIPS -- Official Pytorch Implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis Requirements pip install -r requi

Multimodal Lab @ Samsung AI Center Moscow

201 Dec 21, 2022

Official pytorch implementation of paper

Implementation for paper "Towards the Generalization of Contrastive Self-Supervised Learning"

Related tags

Overview

Contrastive Self-Supervised Learning on CIFAR-10

Paper

Supported methods

Installation

Dependencies

Evaluation

Results

ResNet-18 trained by SimCLR and Barlow Twins over various data augmentation combinations.

ResNet18 trained by SimCLR and Barlow Twins over various color distortion strengths.

Acknowledgement

Citation

You might also like...

Implementation of Nyström Self-attention, from the paper Nyströmformer

Implementation of SETR model, Original paper: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.

Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

Official pytorch implementation of paper "Image-to-image Translation via Hierarchical Style Disentanglement".

PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 2021

Implementation of Barlow Twins paper

Official pytorch implementation of paper "Inception Convolution with Efficient Dilation Search" (CVPR 2021 Oral).

Official implementation of our paper "LLA: Loss-aware Label Assignment for Dense Pedestrian Detection" in Pytorch.

Functional TensorFlow Implementation of Singular Value Decomposition for paper Fast Graph Learning

Releases(v0.2.1)

v0.2.1(Jun 16, 2022)

v0.2.0(Jun 4, 2022)

v0.1.0(Jun 4, 2022)

Owner

Weiran Huang

A Multi-attribute Controllable Generative Model for Histopathology Image Synthesis

PyTorch implementation of paper "StarEnhancer: Learning Real-Time and Style-Aware Image Enhancement" (ICCV 2021 Oral)

Context-Aware Image Matting for Simultaneous Foreground and Alpha Estimation

This repository is the official implementation of Open Rule Induction. This paper has been accepted to NeurIPS 2021.

Log4j JNDI inj. vuln scanner

Minecraft agent to farm resources using reinforcement learning

An executor that loads ONNX models and embeds documents using the ONNX runtime.

A lightweight python AUTOmatic-arRAY library.

CTRL-C: Camera calibration TRansformer with Line-Classification

Implementation of Squeezenet in pytorch, pretrained models on Cifar 10 data to come

This code is an implementation for Singing TTS.

Github project for Attention-guided Temporal Coherent Video Object Matting.

Unpaired Caricature Generation with Multiple Exaggerations

Image processing in Python

Plug-n-Play Reinforcement Learning in Python with OpenAI Gym and JAX

PGPortfolio: Policy Gradient Portfolio, the source code of "A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem"(https://arxiv.org/pdf/1706.10059.pdf).

Gin provides a lightweight configuration framework for Python

The 2nd place solution of 2021 google landmark retrieval on kaggle.

Robotics with GPU computing

A high-performance anchor-free YOLO. Exceeding yolov3~v5 with ONNX, TensorRT, NCNN, and Openvino supported.