TensorFlow implementation of Barlow Twins (Barlow Twins: Self-Supervised Learning via Redundancy Reduction)

Last update: Sep 14, 2022

Overview

Barlow-Twins-TF

This repository implements Barlow Twins (Barlow Twins: Self-Supervised Learning via Redundancy Reduction) in TensorFlow and demonstrates it on the CIFAR10 dataset.

Summary:

With a ResNet20 as a trunk and a 3-layer MLP (each layer containing 2048 units) and 100 epochs of pre-training, this training notebook can give 62.61% accuracy on the CIFAR10 test set. The pre-training total takes ~23 minutes on a single Tesla V100. There are minor differences from the original implementation. However, the original loss function and the other minor details like having a big enough projection dimension have been maintained.

For details on Barlow Twins, I suggest reading the original paper, it's really well-written.

Loss progress during pre-training

Other notes

Pre-trained model is available here.
To follow the original implementation details as closely as possible, a WarmUpCosine learning rate schedule has been used during pre-training:
During linear evaluation, Cosine Decay has been used.

Acknowledgements

Thanks to Stéphane Deny (one of the authors of the paper) for helping me catch a pesky bug.

Releases(v1.0.0)

v1.0.0(May 10, 2021)

The archive contains a ResNet20 encoder (trunk + projection head) trained using Barlow Twins for 100 epochs with CIFAR10.
Source code(tar.gz)
Source code(zip)
barlow_twins.tar.gz(38.07 MB)

Public implementation of "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression" from CoRL'21

Self-Supervised Reward Regression (SSRR) Codebase for CoRL 2021 paper "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression "

19 Dec 12, 2022

An official PyTorch implementation of the TKDE paper "Self-Supervised Graph Representation Learning via Topology Transformations".

Self-Supervised Graph Representation Learning via Topology Transformations This repository is the official PyTorch implementation of the following pap

2 Oct 31, 2022

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

Twins: Revisiting the Design of Spatial Attention in Vision Transformers Very recently, a variety of vision transformer architectures for dense predic

482 Dec 18, 2022

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

Core-tuning This repository is the official implementation of ``Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regular

18 Dec 17, 2022

PyTorch implementation of the paper Deep Networks from the Principle of Rate Reduction

A repository that shares tuning results of trained models generated by TensorFlow / Keras. Post-training quantization (Weight Quantization, Integer Quantization, Full Integer Quantization, Float16 Quantization), Quantization-aware training. TensorFlow Lite. OpenVINO. CoreML. TensorFlow.js. TF-TRT. MediaPipe. ONNX. [.tflite,.h5,.pb,saved_model,tfjs,tftrt,mlmodel,.xml/.bin, .onnx]

PINTO_model_zoo Please read the contents of the LICENSE file located directly under each folder before using the model. My model conversion scripts ar

2.4k Jan 5, 2023

TensorFlow implementation of Barlow Twins (Barlow Twins: Self-Supervised Learning via Redundancy Reduction)

Related tags

Overview

Barlow-Twins-TF

Loss progress during pre-training

Other notes

Acknowledgements

You might also like...

Public implementation of "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression" from CoRL'21

An official PyTorch implementation of the TKDE paper "Self-Supervised Graph Representation Learning via Topology Transformations".

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

PyTorch implementation of the paper Deep Networks from the Principle of Rate Reduction

Official NumPy Implementation of Deep Networks from the Principle of Rate Reduction (2021)

Model-based 3D Hand Reconstruction via Self-Supervised Learning, CVPR2021

TLDR: Twin Learning for Dimensionality Reduction

Releases(v1.0.0)

v1.0.0(May 10, 2021)

Owner

Sayak Paul

The Agriculture Domain of ERPNext comes with features to record crops and land

A minimal implementation of Gaussian process regression in PyTorch

PyTorch implementation of VAGAN: Visual Feature Attribution Using Wasserstein GANs

The Official PyTorch Implementation of "LSGM: Score-based Generative Modeling in Latent Space" (NeurIPS 2021)

PiRank: Learning to Rank via Differentiable Sorting

Selecting Parallel In-domain Sentences for Neural Machine Translation Using Monolingual Texts

Object DGCNN and DETR3D, Our implementations are built on top of MMdetection3D.

Fast Learning of MNL Model From General Partial Rankings with Application to Network Formation Modeling

Capstone-Project-2 - A game program written in the Python language

Various operations like path tracking, counting, etc by using yolov5

Evaluating Cross-lingual Sentence Representations

Regularizing Nighttime Weirdness: Efficient Self-supervised Monocular Depth Estimation in the Dark (ICCV 2021)

The official implementation of CSG-Stump: A Learning Friendly CSG-Like Representation for Interpretable Shape Parsing

TensorFlow implementation of Elastic Weight Consolidation

FSL-Mate: A collection of resources for few-shot learning (FSL).

Code for EmBERT, a transformer model for embodied, language-guided visual task completion.

a delightful machine learning tool that allows you to train, test and use models without writing code

SegNet-Basic with Keras

This is an example of object detection on Micro bacterium tuberculosis using Mask-RCNN

Few-shot NLP benchmark for unified, rigorous eval