[NIPS 2021] UOTA: Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration.

Last update: Jun 29, 2022

Related tags

Deep Learning uota

Overview

UOTA: Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration

This repository is the official PyTorch implementation of UOTA (Unsupervised OuTlier Arbitration).

0 Requirements

Python 3.6
PyTorch install = 1.6.0
torchvision install = 0.7.0
CUDA 10.1
Apex with CUDA extension
Other dependencies: opencv-python, scipy, pandas, numpy

1 Pretraining

We release a demo to pretrain ResNet50 on ImageNet1K with SwAV+UOTA pretrained models.

1.1 SwAV+UOTA pretrain

To train SwAV+UOTA on a single node with 4 gpus for 200 epochs, run:

DATASET_PATH="path/to/ImageNet1K/train"
EXPERIMENT_PATH="path/to/experiment"

python -m torch.distributed.launch --nproc_per_node=4 main_uota.py \
--data_path ${DATASET_PATH} \
--nmb_crops 2 6 \
--size_crops 224 96 \
--min_scale_crops 0.14 0.05 \
--max_scale_crops 1. 0.14 \
--crops_for_assign 0 1 \
--use_pil_blur true \
--epochs 200 \
--warmup_epochs 0 \
--batch_size 64 \
--base_lr 0.6 \
--final_lr 0.0006 \
--uota_tau 350. \
--epoch_uota_starts 100 \
--wd 0.000001 \
--use_fp16 true \
--dist_url "tcp://localhost:40000" \
--arch uota_r50 \
--sync_bn pytorch \
--dump_path ${EXPERIMENT_PATH}

2 Linear Evaluation

To train a linear classifier on frozen features out of deep network pretrained via various self-supervised pretraining methods, run:

DATASET_PATH="path/to/ImageNet1K"
EXPERIMENT_PATH="path/to/experiment"
LINCLS_PATH="path/to/lincls"

python -m torch.distributed.launch --nproc_per_node=4 eval_linear.py \
--data_path ${DATASET_PATH} \
--arch resnet50 \
--lr 1.2 \
--dump_path ${LINCLS_PATH} \
--pretrained ${EXPERIMENT_PATH}/swav_uota_r50_e200_pretrained.pth \
--batch_size 64 \
--num_classes 100 \

3 Results

To compare with SwAV fairly, we provide a SwAV+UOTA model with ResNet-50 architecture pretrained on ImageNet1K for 200 epochs, and release the pretrained model and the linear classier.

method	epochs	batch-size	multi-crop	ImageNet1K top-1 acc.	pretrained model	linear classifier
SwAV	200	256	2x224 + 6x96	72.7	/	/
SwAV + UOTA	200	256	2x224 + 6x96	73.5	pretrained	linear

4 Citation

@InProceedings{wang2021NeurIPS,
  title={Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration},
  author={Wang, Yu and Lin, Jingyang and Zou, Jingjing and Pan, Yingwei and Yao, Ting and Mei, Tao},
  booktitle={NeurIPS},
  year={2021},
}

You might also like...

PyTorch implementation of spectral graph ConvNets, NIPS’16

Graph ConvNets in PyTorch October 15, 2017 Xavier Bresson http://www.ntu.edu.sg/home/xbresson https://github.com/xbresson https://twitter.com/xbresson

287 Jan 4, 2023

PyTorch implementation of the Value Iteration Networks (VIN) (NIPS '16 best paper)

Value Iteration Networks in PyTorch Tamar, A., Wu, Y., Thomas, G., Levine, S., and Abbeel, P. Value Iteration Networks. Neural Information Processing

75 Nov 24, 2022

Pytorch implementation of Value Iteration Networks (NIPS 2016 best paper)

VIN: Value Iteration Networks A quick thank you A few others have released amazing related work which helped inspire and improve my own implementation

297 Dec 26, 2022

pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

Unofficial implementation: MoCo: Momentum Contrast for Unsupervised Visual Representation Learning (Paper) InsDis: Unsupervised Feature Learning via N

16 Nov 4, 2020

The official implementation of CVPR 2021 Paper: Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation.

Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation This repository is the official implementation of CVPR 2021 paper:

9 Nov 14, 2022

(JMLR'19) A Python Toolbox for Scalable Outlier Detection (Anomaly Detection)

[NIPS 2021] UOTA: Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration.

Related tags

Overview

UOTA: Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration

0 Requirements

1 Pretraining

1.1 SwAV+UOTA pretrain

2 Linear Evaluation

3 Results

4 Citation

You might also like...

PyTorch implementation of spectral graph ConvNets, NIPS’16

PyTorch implementation of the Value Iteration Networks (VIN) (NIPS '16 best paper)

Pytorch implementation of Value Iteration Networks (NIPS 2016 best paper)

pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

The official implementation of CVPR 2021 Paper: Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation.

(JMLR'19) A Python Toolbox for Scalable Outlier Detection (Anomaly Detection)

Streaming Anomaly Detection Framework in Python (Outlier Detection for Streaming Data)

A gesture recognition system powered by OpenPose, k-nearest neighbours, and local outlier factor.

Outlier Exposure with Confidence Control for Out-of-Distribution Detection

Releases(v1.0.0)

v1.0.0(Oct 26, 2021)

Owner

Deep Learning pipeline for motor-imagery classification.

Dynamic vae - Dynamic VAE algorithm is used for anomaly detection of battery data

Procedural 3D data generation pipeline for architecture

This is the official source code of "BiCAT: Bi-Chronological Augmentation of Transformer for Sequential Recommendation".

Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes

LERP : Label-dependent and event-guided interpretable disease risk prediction using EHRs

ivadomed is an integrated framework for medical image analysis with deep learning.

Official implementation for paper: A Latent Transformer for Disentangled Face Editing in Images and Videos.

Efficient Two-Step Networks for Temporal Action Segmentation (Neurocomputing 2021)

[NeurIPS'21 Spotlight] PyTorch code for our paper "Aligned Structured Sparsity Learning for Efficient Image Super-Resolution"

Improving Contrastive Learning by Visualizing Feature Transformation, ICCV 2021 Oral

Neighbor2Seq: Deep Learning on Massive Graphs by Transforming Neighbors to Sequences

Alfred-Restore-Iterm-Arrangement - An Alfred workflow to restore iTerm2 window Arrangements

My solution for the 7th place / 245 in the Umoja Hack 2022 challenge

Speeding-Up Back-Propagation in DNN: Approximate Outer Product with Memory

Underwater industrial application yolov5m6

A simple python module to generate anchor (aka default/prior) boxes for object detection tasks.

Clustergram - Visualization and diagnostics for cluster analysis in Python

Code for Massive-scale Decoding for Text Generation using Lattices