Multilingual word vectors in 78 languages

Overview

Aligning the fastText vectors of 78 languages

Facebook recently open-sourced word vectors in 89 languages. However these vectors are monolingual; meaning that while similar words within a language share similar vectors, translation words from different languages do not have similar vectors. In a recent paper at ICLR 2017, we showed how the SVD can be used to learn a linear transformation (a matrix), which aligns monolingual vectors from two languages in a single vector space. In this repository we provide 78 matrices, which can be used to align the majority of the fastText languages in a single space.

This readme explains how the matrices should be used. We also present a simple evaluation task, where we show we are able to successfully predict the translations of words in multiple languages. Our procedure relies on collecting bilingual training dictionaries of word pairs in two languages, but remarkably we are able to successfully predict the translations of words between language pairs for which we had no training dictionary!

Word embeddings define the similarity between two words by the normalised inner product of their vectors. The matrices in this repository place languages in a single space, without changing any of these monolingual similarity relationships. When you use the resulting multilingual vectors for monolingual tasks, they will perform exactly the same as the original vectors. To learn more about word embeddings, check out Colah's blog or Sam's introduction to vector representations.

Note that since we released this repository Facebook have released an additional 204 languages; however the word vectors of the original 90 languages have not changed, and the transformations provided in this repository will still work. If you would like to learn your own alignment matrices, we provide an example in align_your_own.ipynb.

If you use this repository, please cite:

Offline bilingual word vectors, orthogonal transformations and the inverted softmax
Samuel L. Smith, David H. P. Turban, Steven Hamblin and Nils Y. Hammerla
ICLR 2017 (conference track)

TLDR, just tell me what to do!

Clone a local copy of this repository, and download the fastText vectors you need from here. I'm going to assume you've downloaded the vectors for French and Russian in the text format. Let's say we want to compare the similarity of "chat" and "кот". We load the word vectors:

from fasttext import FastVector
fr_dictionary = FastVector(vector_file='wiki.fr.vec')
ru_dictionary = FastVector(vector_file='wiki.ru.vec')

We can extract the word vectors and calculate their cosine similarity:

fr_vector = fr_dictionary["chat"]
ru_vector = ru_dictionary["кот"]
print(FastVector.cosine_similarity(fr_vector, ru_vector))
# Result should be 0.02

The cosine similarity runs between -1 and 1. It seems that "chat" and "кот" are neither similar nor dissimilar. But now we apply the transformations to align the two dictionaries in a single space:

fr_dictionary.apply_transform('alignment_matrices/fr.txt')
ru_dictionary.apply_transform('alignment_matrices/ru.txt')

And re-evaluate the cosine similarity:

print(FastVector.cosine_similarity(fr_dictionary["chat"], ru_dictionary["кот"]))
# Result should be 0.43

Turns out "chat" and "кот" are pretty similar after all. This is good, since they both mean "cat".

Ok, so how did you obtain these matrices?

Of the 89 languages provided by Facebook, 78 are supported by the Google Translate API. We first obtained the 10,000 most common words in the English fastText vocabulary, and then use the API to translate these words into the 78 languages available. We split this vocabulary in two, assigning the first 5000 words to the training dictionary, and the second 5000 to the test dictionary.

We described the alignment procedure in this blog. It takes two sets of word vectors and a small bilingual dictionary of translation pairs in two languages; and generates a matrix which aligns the source language with the target. Sometimes Google translates an English word to a non-English phrase, in these cases we average the word vectors contained in the phrase.

To place all 78 languages in a single space, we align every language to the English vectors (the English matrix is the identity).

Right, now prove that this procedure actually worked...

To prove that the procedure works, we can predict the translations of words not seen in the training dictionary. For simplicity we predict translations by nearest neighbours. So for example, if we wanted to translate "dog" into Swedish, we would simply find the Swedish word vector whose cosine similarity to the "dog" word vector is highest.

First things first, let's test the translation performance from English into every other language. For each language pair, we extract a set of 2500 word pairs from the test dictionary. The precision @n denotes the probability that, of the 2500 target words in this set, the true translation was one of the top n nearest neighbours of the source word. If the alignment was completely random, we would expect the precision @1 to be around 0.0004.

Target language Precision @1 Precision @5 Precision @10
fr 0.73 0.86 0.88
pt 0.73 0.86 0.89
es 0.72 0.85 0.88
it 0.70 0.86 0.89
nl 0.68 0.83 0.86
no 0.68 0.85 0.89
da 0.66 0.84 0.88
ca 0.66 0.81 0.86
sv 0.65 0.82 0.86
cs 0.64 0.81 0.85
ro 0.63 0.81 0.85
de 0.62 0.75 0.78
pl 0.62 0.79 0.83
hu 0.61 0.80 0.84
fi 0.61 0.80 0.84
eo 0.61 0.80 0.85
ru 0.60 0.78 0.82
gl 0.60 0.77 0.82
mk 0.58 0.79 0.84
id 0.58 0.81 0.86
bg 0.57 0.77 0.82
ms 0.57 0.81 0.86
uk 0.57 0.75 0.79
sh 0.56 0.77 0.81
hr 0.56 0.75 0.80
tr 0.56 0.77 0.81
sl 0.56 0.77 0.82
el 0.54 0.75 0.80
sk 0.54 0.75 0.81
et 0.53 0.73 0.78
sr 0.53 0.72 0.77
af 0.52 0.75 0.80
lt 0.50 0.72 0.79
ar 0.48 0.69 0.75
bs 0.47 0.70 0.77
lv 0.47 0.68 0.75
eu 0.46 0.68 0.75
fa 0.45 0.68 0.75
hy 0.43 0.66 0.73
sq 0.43 0.65 0.71
be 0.43 0.64 0.70
zh 0.40 0.68 0.75
ka 0.40 0.63 0.71
cy 0.39 0.63 0.71
hi 0.39 0.58 0.63
az 0.38 0.60 0.67
ko 0.37 0.58 0.66
te 0.36 0.56 0.63
kk 0.35 0.60 0.68
he 0.33 0.45 0.48
fy 0.33 0.52 0.60
vi 0.31 0.53 0.62
ta 0.31 0.50 0.56
bn 0.30 0.49 0.56
ur 0.29 0.52 0.61
is 0.29 0.51 0.59
tl 0.28 0.51 0.59
kn 0.28 0.43 0.46
gu 0.25 0.44 0.51
mn 0.25 0.49 0.58
uz 0.24 0.43 0.51
si 0.22 0.40 0.45
ml 0.21 0.35 0.39
ky 0.20 0.40 0.49
mr 0.20 0.37 0.44
th 0.20 0.33 0.38
la 0.19 0.34 0.42
ja 0.18 0.44 0.56
ne 0.16 0.33 0.38
pa 0.16 0.32 0.38
tg 0.14 0.31 0.39
km 0.12 0.26 0.30
my 0.10 0.19 0.23
lb 0.09 0.18 0.21
mg 0.07 0.18 0.25
ceb 0.06 0.13 0.18

As you can see, the alignment is consistently much better than random! In general, the procedure works best for other European languages like French, Portuguese and Spanish. We use 2500 word pairs, because of the 5000 words in the test dictionary, not all the words found by the Google Translate API are actually present in the fastText vocabulary.

Now let's do something much more exciting, let's evaluate the translation performance between all possible language pairs. We exhibit this translation performance on the heatmap below, where the colour of an element denotes the precision @1 when translating from the language of the row into the language of the column.

Cool huh!

We should emphasize that all of the languages were aligned to English only. We did not provide training dictionaries between non-English language pairs. Yet we are still able to succesfully predict translations between pairs of non-English languages remarkably accurately.

We expect the diagonal elements of the matrix above to be 1, since a language should translate perfectly to itself. However in practice this does not always occur, because we constructed the training and test dictionaries by translating common English words into the other languages. Sometimes multiple English words translate to the same non-English word, and so the same non-English word may appear multiple times in the test set. We haven't properly accounted for this, which reduces the translation performance.

Intriquingly, even though we only directly aligned the languages to English, sometimes a language translates better to another non-English language than it does to English! We can calculate the inter-pair precision of two languages; the average precision from language 1 to language 2 and vice versa. We can also calculate the English-pair precision; the average of the precision from English to language 1 and from English to language 2. Below we list all the language pairs for which the inter-pair precision exceeds the English-pair precision:

Language 1 Language 2 Inter-pair precision @1 English-pair precision @1
bs sh 0.88 0.52
ru uk 0.84 0.58
ca es 0.82 0.69
cs sk 0.82 0.59
hr sh 0.78 0.56
be uk 0.77 0.50
gl pt 0.76 0.66
bs hr 0.74 0.52
be ru 0.73 0.51
da no 0.73 0.67
sr sh 0.73 0.54
pt es 0.72 0.72
ca pt 0.70 0.69
gl es 0.70 0.66
hr sr 0.69 0.54
ca gl 0.68 0.63
bs sr 0.67 0.50
mk sr 0.56 0.55
kk ky 0.30 0.28

All of these language pairs share very close linguistic roots. For instance the first pair above are Bosnian and Serbo-Croatian; Bosnian is a variant of Serbo-Croatian. The second pair is Russian and Ukranian; both east-slavic languages. It seems that the more similar two languages are, the more similar the geometry of their fastText vectors; leading to improved translation performance.

How do I know these matrices don't change the monolingual vectors?

The matrices provided in this repository are orthogonal. Intuitively, each matrix can be broken down into a series of rotations and reflections. Rotations and reflections do not change the distance between any two points in a vector space; and consequently none of the inner products between word vectors within a language are changed, only the inner products between the word vectors of different languages are affected.

References

There are a number of great papers on this topic. We've listed a few of them below:

  1. Enriching word vectors with subword information
    Bojanowski et al., 2016
  2. Offline bilingual word vectors, orthogonal transformations and the inverted softmax
    Smith et al., ICLR 2017
  3. Exploiting similarities between languages for machine translation
    Mikolov et al., 2013
  4. Improving vector space word representations using multilingual correlation
    Faruqui and Dyer, EACL 2014
  5. Improving zero-shot learning by mitigating the hubness problem
    Dinu et al., 2014
  6. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance
    Artetxe et al., EMNLP 2016

Training and test dictionaries

A number of readers have expressed an interest in the training and test dictionaries we used in this repository. We would have liked to upload these, however, while we have not taken legal advice, we are concerned that this could be interpreted as breaking the terms of the Google Translate API.

License

The transformation matrices are distributed under the Creative Commons Attribution-Share-Alike License 3.0.

Owner
Babylon Health
Putting an accessible and affordable health service in the hands of every person on earth.
Babylon Health
Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System

Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System Authors: Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai

Amazon Web Services - Labs 124 Jan 03, 2023
chaii - hindi & tamil question answering

chaii - hindi & tamil question answering This is the solution for rank 5th in Kaggle competition: chaii - Hindi and Tamil Question Answering. The comp

abhishek thakur 33 Dec 18, 2022
Unofficial Python library for using the Polish Wordnet (plWordNet / Słowosieć)

Polish Wordnet Python library Simple, easy-to-use and reasonably fast library for using the Słowosieć (also known as PlWordNet) - a lexico-semantic da

Max Adamski 12 Dec 23, 2022
Download videos from YouTube/Twitch/Twitter right in the Windows Explorer, without installing any shady shareware apps

youtube-dl and ffmpeg Windows Explorer Integration Download videos from YouTube/Twitch/Twitter and more (any platform that is supported by youtube-dl)

Wolfgang 226 Dec 30, 2022
Some embedding layer implementation using ivy library

ivy-manual-embeddings Some embedding layer implementation using ivy library. Just for fun. It is based on NYCTaxiFare dataset from kaggle (cut down to

Ishtiaq Hussain 2 Feb 10, 2022
A Fast Command Analyser based on Dict and Pydantic

Alconna Alconna 隶属于ArcletProject, 在Cesloi内有内置 Alconna 是 Cesloi-CommandAnalysis 的高级版,支持解析消息链 一般情况下请当作简易的消息链解析器/命令解析器 文档 暂时的文档 Example from arclet.alcon

19 Jan 03, 2023
Installation, test and evaluation of Scribosermo speech-to-text engine

Scribosermo STT Setup Scribosermo is a LGPL licensed, open-source speech recognition engine to "Train fast Speech-to-Text networks in different langua

Florian Quirin 3 Jun 20, 2022
Using context-free grammar formalism to parse English sentences to determine their structure to help computer to better understand the meaning of the sentence.

Sentance Parser Executing the Program Make sure Python 3.6+ is installed. Install requirements $ pip install requirements.txt Run the program:

Vaibhaw 12 Sep 28, 2022
A fast, efficient universal vector embedding utility package.

Magnitude: a fast, simple vector embedding utility library A feature-packed Python package and vector storage file format for utilizing vector embeddi

Plasticity 1.5k Jan 02, 2023
A modular Karton Framework service that unpacks common packers like UPX and others using the Qiling Framework.

Unpacker Karton Service A modular Karton Framework service that unpacks common packers like UPX and others using the Qiling Framework. This project is

c3rb3ru5 45 Jan 05, 2023
This repository contains examples of Task-Informed Meta-Learning

Task-Informed Meta-Learning This repository contains examples of Task-Informed Meta-Learning (paper). We consider two tasks: Crop Type Classification

10 Dec 19, 2022
scikit-learn wrappers for Python fastText.

skift scikit-learn wrappers for Python fastText. from skift import FirstColFtClassifier df = pandas.DataFrame([['woof', 0], ['meow', 1]], colu

Shay Palachy 233 Sep 09, 2022
Lingtrain Aligner — ML powered library for the accurate texts alignment.

Lingtrain Aligner ML powered library for the accurate texts alignment in different languages. Purpose Main purpose of this alignment tool is to build

Sergei Averkiev 76 Dec 14, 2022
A telegram bot to translate 100+ Languages

🔥 GOOGLE TRANSLATER 🔥 The owner would not be responsible for any kind of bans due to the bot. • ⚡ INSTALLING ⚡ • • 🔰 Deploy To Railway 🔰 • • ✅ OFF

Aɴᴋɪᴛ Kᴜᴍᴀʀ 5 Dec 20, 2021
An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition

CRNN paper:An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition 1. create your ow

Tsukinousag1 3 Apr 02, 2022
Unlimited Call - Text Bombing Tool

FastBomber Unlimited Call - Text Bombing Tool Installation On Termux

Aryan 6 Nov 10, 2022
Findings of ACL 2021

Assessing Dialogue Systems with Distribution Distances [arXiv][code] We propose to measure the performance of a dialogue system by computing the distr

Yahui Liu 16 Feb 24, 2022
ProtFeat is protein feature extraction tool that utilizes POSSUM and iFeature.

Description: ProtFeat is designed to extract the protein features by employing POSSUM and iFeature python-based tools. ProtFeat includes a total of 39

GOKHAN OZSARI 5 Dec 16, 2022
A programming language with logic of Python, and syntax of all languages.

Pytov The idea was to take all well known syntaxes, and combine them into one programming language with many posabilities. Installation Install using

Yuval Rosen 14 Dec 07, 2022
A Python script that compares files in directories

compare-files A Python script that compares files in different directories, this is similar to the command filecmp.cmp(f1, f2). I made this script in

Colvin 1 Oct 15, 2021