Implementation of the paper "Fine-Tuning Transformers: Vocabulary Transfer"

Last update: Nov 30, 2022

Related tags

Overview

Transformer-vocabulary-transfer

Implementation of the paper "Fine-Tuning Transformers: Vocabulary Transfer"

Description

step 1 - Create SentencePiece vocabulary for dataset
step 2 - Train the first level model (BertForMaskedLM) on English Wikipedia from scratch
step 3 - Match vocabulary (first level model dataset & downstream task dataset)
step 4 - Transfer dictionary using mapping. Сreate folders and raw models for experiments.
step 5 - Train 1 epoch BertForMaskedLM on downstream task
step 6 - Train final (BertForSequenceClassification) downstream model

Citation

I. Samenko, A. Tikhonov, B. Kozlovsky, I. P. Yamshchikov. Fine-Tuning Transformers: Vocabulary Transfer.

Contact

*Igor Samenko: [email protected]

Owner

LEYA

LEYA is a Natural Language Processing research laboratory co-founded by Yandex and Higher School of Economics in Saint-Petersburg.

GitHub Repository

Hyperopt for solving CIFAR-100 with a convolutional neural network (CNN) built with Keras and TensorFlow, GPU backend

Hyperopt for solving CIFAR-100 with a convolutional neural network (CNN) built with Keras and TensorFlow, GPU backend This project acts as both a tuto

103 Jul 22, 2022

Visual Question Answering in Pytorch

Visual Question Answering in pytorch /!\ New version of pytorch for VQA available here: https://github.com/Cadene/block.bootstrap.pytorch This repo wa

672 Jan 01, 2023

PyTorch implementation of Munchausen Reinforcement Learning based on DQN and SAC. Handles discrete and continuous action spaces

Exploring Munchausen Reinforcement Learning This is the project repository of my team in the "Advanced Deep Learning for Robotics" course at TUM. Our

10 Mar 10, 2022

NeRF visualization library under construction

NeRF visualization library using PlenOctrees, under construction pip install nerfvis Docs will be at: https://nerfvis.readthedocs.org import nerfvis s

196 Jan 04, 2023

Sign-to-Speech for Sign Language Understanding: A case study of Nigerian Sign Language

Sign-to-Speech for Sign Language Understanding: A case study of Nigerian Sign Language This repository contains the code, model, and deployment config

16 Oct 23, 2022

PyTorch implementation of the Pose Residual Network (PRN)

Pose Residual Network This repository contains a PyTorch implementation of the Pose Residual Network (PRN) presented in our ECCV 2018 paper: Muhammed

289 Nov 28, 2022

Code for paper "A Critical Assessment of State-of-the-Art in Entity Alignment" (https://arxiv.org/abs/2010.16314)

A Critical Assessment of State-of-the-Art in Entity Alignment This repository contains the source code for the paper A Critical Assessment of State-of

16 Oct 14, 2022

Bio-OFC gym implementation and Gym-Fly environment

Bio-OFC gym implementation and Gym-Fly environment This repository includes the gym compatible implementation of the Bio-OFC algorithm from the paper

1 Nov 16, 2021

PyTorch implementation for our paper Learning Character-Agnostic Motion for Motion Retargeting in 2D, SIGGRAPH 2019

Learning Character-Agnostic Motion for Motion Retargeting in 2D We provide PyTorch implementation for our paper Learning Character-Agnostic Motion for

367 Dec 22, 2022

Instantaneous Motion Generation for Robots and Machines.

Ruckig Instantaneous Motion Generation for Robots and Machines. Ruckig generates trajectories on-the-fly, allowing robots and machines to react instan

374 Dec 23, 2022

LIMEcraft: Handcrafted superpixel selectionand inspection for Visual eXplanations

LIMEcraft LIMEcraft: Handcrafted superpixel selectionand inspection for Visual eXplanations The LIMEcraft algorithm is an explanatory method based on

4 Aug 01, 2022

Official implementation for the paper "Attentive Prototypes for Source-free Unsupervised Domain Adaptive 3D Object Detection"

Attentive Prototypes for Source-free Unsupervised Domain Adaptive 3D Object Detection PyTorch code release of the paper "Attentive Prototypes for Sour

23 Oct 17, 2022

Implementation of the paper "Fine-Tuning Transformers: Vocabulary Transfer"

Related tags

Overview

Transformer-vocabulary-transfer

Description

Citation

Contact

Owner

LEYA

Hyperopt for solving CIFAR-100 with a convolutional neural network (CNN) built with Keras and TensorFlow, GPU backend

Visual Question Answering in Pytorch

PyTorch implementation of Munchausen Reinforcement Learning based on DQN and SAC. Handles discrete and continuous action spaces

NeRF visualization library under construction

Sign-to-Speech for Sign Language Understanding: A case study of Nigerian Sign Language

PyTorch implementation of the Pose Residual Network (PRN)

Code for paper "A Critical Assessment of State-of-the-Art in Entity Alignment" (https://arxiv.org/abs/2010.16314)

Bio-OFC gym implementation and Gym-Fly environment

PyTorch implementation for our paper Learning Character-Agnostic Motion for Motion Retargeting in 2D, SIGGRAPH 2019

Instantaneous Motion Generation for Robots and Machines.

LIMEcraft: Handcrafted superpixel selectionand inspection for Visual eXplanations

Official implementation for the paper "Attentive Prototypes for Source-free Unsupervised Domain Adaptive 3D Object Detection"

Simple node deletion tool for onnx.

[NeurIPS 2021 Spotlight] Code for Learning to Compose Visual Relations

Supervised multi-SNE (S-multi-SNE): Multi-view visualisation and classification

NAS-FCOS: Fast Neural Architecture Search for Object Detection (CVPR 2020)

Level Based Customer Segmentation

PointPillars inference with TensorRT

Existing Literature about Machine Unlearning

Python tools for 3D face: 3DMM, Mesh processing(transform, camera, light, render), 3D face representations.