BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

Hl classification bc - A Network-Based High-Level Data Classification Algorithm Using Betweenness Centrality

UDP++ (ECCVW 2020 Oral), (Winner of COCO 2020 Keypoint Challenge).

Code and real data for the paper "Counterfactual Temporal Point Processes", available at arXiv.

Code for the paper: Audio-Visual Scene Analysis with Self-Supervised Multisensory Features

Pytorch Lightning Distributed Accelerators using Ray

The codes and related files to reproduce the results for Image Similarity Challenge Track 1.

People Interaction Graph

[MICCAI'20] AlignShift: Bridging the Gap of Imaging Thickness in 3D Anisotropic Volumes

Powerful and efficient Computer Vision Annotation Tool (CVAT)

Classification Modeling: Probability of Default

My tensorflow implementation of "A neural conversational model", a Deep learning based chatbot

Newt - a Gaussian process library in JAX.

Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Public repository created to store my custom-made tools for Just Dance (UbiArt Engine)

Transparent Transformer Segmentation

Generate images from texts. In Russian. In PaddlePaddle

A tool to visualise the results of AlphaFold2 and inspect the quality of structural predictions

naked is a Python tool which allows you to strip a model and only keep what matters for making predictions.

GLaRA: Graph-based Labeling Rule Augmentation for Weakly Supervised Named Entity Recognition

Towards Flexible Blind JPEG Artifacts Removal (FBCNN, ICCV 2021)