Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Last update: Dec 29, 2022

Related tags

Deep Learning wechsel

Overview

WECHSEL

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

arXiv: https://arxiv.org/abs/2112.06598

Models from the paper are available on the HuggingFace Hub:

Installation

We distribute a Python Package via PyPI:

pip install wechsel

Alternatively, clone the repository, install requirements.txt and run the code in wechsel/.

Example usage

Transferring English roberta-base to Swahili:

import torch
from transformers import AutoModel, AutoTokenizer
from datasets import load_dataset
from wechsel import WECHSEL, load_embeddings

source_tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model = AutoModel.from_pretrained("roberta-base")

target_tokenizer = source_tokenizer.train_new_from_iterator(
    load_dataset("oscar", "unshuffled_deduplicated_sw", split="train")["text"],
    vocab_size=len(source_tokenizer)
)

wechsel = WECHSEL(
    load_embeddings("en"),
    load_embeddings("sw"),
    bilingual_dictionary="swahili"
)

target_embeddings, info = wechsel.apply(
    source_tokenizer,
    target_tokenizer,
    model.get_input_embeddings().weight.detach().numpy(),
)

model.get_input_embeddings().weight.data = torch.from_numpy(target_embeddings)

# use `model` and `target_tokenizer` to continue training in Swahili!

Bilingual dictionaries

We distribute 3276 bilingual dictionaries from English to other languages for use with WECHSEL in dicts/.

Citation

Please cite WECHSEL as

@misc{minixhofer2021wechsel,
      title={WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models}, 
      author={Benjamin Minixhofer and Fabian Paischer and Navid Rekabsaz},
      year={2021},
      eprint={2112.06598},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Related tags

Overview

WECHSEL

Installation

Example usage

Bilingual dictionaries

Citation

Owner

Institute of Computational Perception

DeFMO: Deblurring and Shape Recovery of Fast Moving Objects (CVPR 2021)

StrongSORT: Make DeepSORT Great Again

Relative Uncertainty Learning for Facial Expression Recognition

A Kernel fuzzer focusing on race bugs

The official homepage of the (outdated) COCO-Stuff 10K dataset.

Incorporating Transformer and LSTM to Kalman Filter with EM algorithm

Face Mask Detection system based on computer vision and deep learning using OpenCV and Tensorflow/Keras

DECA: Detailed Expression Capture and Animation (SIGGRAPH 2021)

A toolkit for Lagrangian-based constrained optimization in Pytorch

LaneAF: Robust Multi-Lane Detection with Affinity Fields

Meta-learning for NLP

The 1st place solution of track2 (Vehicle Re-Identification) in the NVIDIA AI City Challenge at CVPR 2021 Workshop.

Toolkit for collecting and applying prompts

AI virtual gym is an AI program which can be used to exercise and can be used to see if we are doing the exercises

Pytorch implementation of TailCalibX : Feature Generation for Long-tail Classification

DecoupledNet is semantic segmentation system which using heterogeneous annotations

A simple and extensible library to create Bayesian Neural Network layers on PyTorch.

Computationally efficient algorithm that identifies boundary points of a point cloud.

AI-UPV at IberLEF-2021 DETOXIS task: Toxicity Detection in Immigration-Related Web News Comments Using Transformers and Statistical Models

Differentiable Annealed Importance Sampling (DAIS)