Beyond Masking: Demystifying Token-Based Pre-Training for Vision Transformers

Last update: Sep 27, 2022

Related tags

Text Data & NLP beyond_masking

Overview

beyond masking

Beyond Masking: Demystifying Token-Based Pre-Training for Vision Transformers

The code is coming

Figure 1: Pipeline of token-based pre-training.

Figure 2: The visualization of the proposed 5 tasks.

main results

All the results are pre-trained for 300 epochs using Vit-base as default.

	zoomed-in	zoomed-out	distorted	blurred	de-colorized
finetune	`82.7`	`82.5`	`82.1`	`81.8`	`81.4`

	zoomed-in (a)	mask (m)	(a)+(m)
finetune	`82.7`	`82.9`	`83.2`

We note that the integrated version dose not require extra computational cost.

Effiencicy

Figure 3: Efficiency of the integrated task.

Owner

Yunjie Tian

I am a Ph.D student in University of Chinese Academy of Sciences (UCAS), studying computer vision, especially NAS and self-supervised learning.

GitHub Repository

Modeling cumulative cases of Covid-19 in the US during the Covid 19 Delta wave using Bayesian methods.

Introduction The goal of this analysis is to find a model that fits the observed cumulative cases of COVID-19 in the US, starting in Mid-July 2021 and

1 Jan 05, 2022

A raytrace framework using taichi language

ti-raytrace The code use Taichi programming language Current implement acceleration lvbh disney brdf How to run First config your anaconda workspace,

73 Dec 11, 2022

Finding Label and Model Errors in Perception Data With Learned Observation Assertions

Finding Label and Model Errors in Perception Data With Learned Observation Assertions This is the project page for Finding Label and Model Errors in P

17 Oct 14, 2022

DeLighT: Very Deep and Light-Weight Transformers

DeLighT: Very Deep and Light-weight Transformers This repository contains the source code of our work on building efficient sequence models: DeFINE (I

440 Dec 18, 2022

Library for Russian imprecise rhymes generation

TOM RHYMER Library for Russian imprecise rhymes generation. Quick Start Generate rhymes by any given rhyme scheme (aabb, abab, aaccbb, etc ...): from

6 Oct 18, 2022

FireFlyer Record file format, writer and reader for DL training samples.

FFRecord The FFRecord format is a simple format for storing a sequence of binary records developed by HFAiLab, which supports random access and Linux

77 Jan 04, 2023

The official implementation of VAENAR-TTS, a VAE based non-autoregressive TTS model.

VAENAR-TTS This repo contains code accompanying the paper "VAENAR-TTS: Variational Auto-Encoder based Non-AutoRegressive Text-to-Speech Synthesis". Sa

138 Oct 28, 2022

Contains analysis of trends from Fitbit Dataset (source: Kaggle) to see how the trends can be applied to Bellabeat customers and Bellabeat products

Contains analysis of trends from Fitbit Dataset (source: Kaggle) to see how the trends can be applied to Bellabeat customers and Bellabeat products.

2 Jan 12, 2022

Samantha, A covid-19 information bot which will provide basic information about this pandemic in form of conversation.

Covid-19-BOT Samantha, A covid-19 information bot which will provide basic information about this pandemic in form of conversation. This bot uses torc

2 Nov 05, 2021

PyTorch source code of NAACL 2019 paper "An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models"

This repository contains source code for NAACL 2019 paper "An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models" (P

89 Aug 12, 2022

Beyond Masking: Demystifying Token-Based Pre-Training for Vision Transformers

Related tags

Overview

beyond masking

main results

Effiencicy

Owner

Yunjie Tian

Modeling cumulative cases of Covid-19 in the US during the Covid 19 Delta wave using Bayesian methods.

A raytrace framework using taichi language

Finding Label and Model Errors in Perception Data With Learned Observation Assertions

DeLighT: Very Deep and Light-Weight Transformers

Library for Russian imprecise rhymes generation

FireFlyer Record file format, writer and reader for DL training samples.

The official implementation of VAENAR-TTS, a VAE based non-autoregressive TTS model.

Contains analysis of trends from Fitbit Dataset (source: Kaggle) to see how the trends can be applied to Bellabeat customers and Bellabeat products

Samantha, A covid-19 information bot which will provide basic information about this pandemic in form of conversation.

PyTorch source code of NAACL 2019 paper "An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models"

Telegram AI chat bot written in Python using Pyrogram

Syntax-aware Multi-spans Generation for Reading Comprehension (TASLP 2022)

Materials (slides, code, assignments) for the NYU class I teach on NLP and ML Systems (Master of Engineering).

Implementation of COCO-LM, Correcting and Contrasting Text Sequences for Language Model Pretraining, in Pytorch

SimCTG - A Contrastive Framework for Neural Text Generation

Sentiment Classification using WSD, Maximum Entropy & Naive Bayes Classifiers

An easy-to-use framework for BERT models, with trainers, various NLP tasks and detailed annonations

Machine translation models released by the Gourmet project

Conditional Transformer Language Model for Controllable Generation

A toolkit for document-level event extraction, containing some SOTA model implementations