Pre-trained BERT Models for Ancient and Medieval Greek, and associated code for LaTeCH 2021 paper titled - "A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval Greek"

Last update: Dec 08, 2022

Related tags

Deep Learning Ancient-Greek-BERT

Overview

Ancient Greek BERT

The first and only available Ancient Greek sub-word BERT model!

State-of-the-art post fine-tuning on Part-of-Speech Tagging and Morphological Analysis.

Pre-trained weights are made available for a standard 12 layer, 768d BERT-base model.

You can also use the model directly on the HuggingFace Model Hub here.

Please refer to our paper titled: "A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval Greek". In Proceedings of The 5th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2021).

How to use

Requirements:

pip install transformers
pip install unicodedata
pip install flair

Can be directly used from the HuggingFace Model Hub with:

from transformers import AutoTokenizer, AutoModel
tokeniser = AutoTokenizer.from_pretrained("pranaydeeps/Ancient-Greek-BERT")
model = AutoModel.from_pretrained("pranaydeeps/Ancient-Greek-BERT")

Fine-tuning for POS/Morphological Analysis

finetune_pos.py can be used to finetune the BERT model for POS tagging on your own data. We provide sample files from the Gold standard treebanks, however the full treebanks can't be made available at this time. Please contact the authors for more details.
Even though the full treebanks aren't made available, we provide a pre-trained POS Tagging model in the directory SuperPeitho-FLAIR-v2, which can directly be used for inference and has an accuracy of ~90 percent on the 3 treebanks available. You can import the pre-trained model in FLAIR with:

from flair.models import SequenceTagger
tagger = SequenceTagger.load('SuperPeitho-FLAIR-v2/final-model.pt')

Training data

The model was initialised from AUEB NLP Group's Greek BERT and subsequently trained on monolingual data from the First1KGreek Project, Perseus Digital Library, PROIEL Treebank and Gorman's Treebank

Training and Eval details

Standard de-accentuating and lower-casing for Greek as suggested in AUEB NLP Group's Greek BERT. The model was trained on 4 NVIDIA Tesla V100 16GB GPUs for 80 epochs, with a max-seq-len of 512 and results in a perplexity of 4.8 on the held out test set. It also gives state-of-the-art results when fine-tuned for PoS Tagging and Morphological Analysis on all 3 treebanks averaging >90% accuracy. Please consult our paper or contact me for further questions!

Cite

If you end up using Ancient-Greek-BERT in your research, please cite the paper:

@inproceedings{ancient-greek-bert,
author = {Singh, Pranaydeep and Rutten, Gorik and Lefever, Els},
title = {A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval Greek},
year = {2021},
booktitle = {The 5th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2021)}
}

Comments

Morphological Analysis Examples
@pranaydeeps Is possible to add simple code example how to do morphological analysis of some short ancient greek sentence? Maybe some examples of:

POS tagging

cosine sentence similarity

Thank You very much for the model. I tried to train smaller BERT model but i didn't have enough GPU resources. I would like to use your model for New Testament analysis.
documentation
opened by PeterPirog 3

source code and pre-trained/fine-tuned checkpoint for NAACL 2021 paper LightningDOT

LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval This repository contains source code and pre-trained/fine-tun

65 Dec 26, 2022

The source codes for ACL 2021 paper 'BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data'

BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data This repository provides the implementation details for

124 Dec 27, 2022

Chinese clinical named entity recognition using pre-trained BERT model

Chinese clinical named entity recognition (CNER) using pre-trained BERT model Introduction Code for paper Chinese clinical named entity recognition wi

109 Dec 14, 2022

Pytorch implementation of our paper under review — Lottery Jackpots Exist in Pre-trained Models

Lottery Jackpots Exist in Pre-trained Models (Paper Link) Requirements Python = 3.7.4 Pytorch = 1.6.1 Torchvision = 0.4.1 Reproduce the Experiment

27 Jun 28, 2022

The implementation of our CIKM 2021 paper titled as: "Cross-Market Product Recommendation"

FOREC: A Cross-Market Recommendation System This repository provides the implementation of our CIKM 2021 paper titled as "Cross-Market Product Recomme

16 Sep 12, 2022

Ever felt tired after preprocessing the dataset, and not wanting to write any code further to train your model? Ever encountered a situation where you wanted to record the hyperparameters of the trained model and able to retrieve it afterward? Models Playground is here to help you do that. Models playground allows you to train your models right from the browser.

Models Playground 🗂️ Upload a Preprocessed Dataset 🌠 Choose whether to perform Classification or Regression 🦹 Enter the Dependent Variable ?

19 Dec 10, 2022

This repo contains the official code and pre-trained models for the Dynamic Vision Transformer (DVT).

Dynamic-Vision-Transformer (Pytorch) This repo contains the official code and pre-trained models for the Dynamic Vision Transformer (DVT). Not All Ima

210 Dec 18, 2022

Code and pre-trained models for MultiMAE: Multi-modal Multi-task Masked Autoencoders

MultiMAE: Multi-modal Multi-task Masked Autoencoders Roman Bachmann*, David Mizrahi*, Andrei Atanov, Amir Zamir Website | arXiv | BibTeX Official PyTo

Visual Intelligence & Learning Lab, Swiss Federal Institute of Technology (EPFL)

385 Jan 6, 2023

Code for reproducing our analysis in the paper titled: Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency

Image Crop Analysis This is a repo for the code used for reproducing our Image Crop Analysis paper as shared on our blog post. If you plan to use this

239 Jan 2, 2023

Pre-trained BERT Models for Ancient and Medieval Greek, and associated code for LaTeCH 2021 paper titled - "A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval Greek"

Related tags

Overview

Ancient Greek BERT

How to use

Fine-tuning for POS/Morphological Analysis

Training data

Training and Eval details

Cite

You might also like...

source code and pre-trained/fine-tuned checkpoint for NAACL 2021 paper LightningDOT

The source codes for ACL 2021 paper 'BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data'

Chinese clinical named entity recognition using pre-trained BERT model

Pytorch implementation of our paper under review — Lottery Jackpots Exist in Pre-trained Models

The implementation of our CIKM 2021 paper titled as: "Cross-Market Product Recommendation"

This repo contains the official code and pre-trained models for the Dynamic Vision Transformer (DVT).

Code and pre-trained models for MultiMAE: Multi-modal Multi-task Masked Autoencoders

Code for reproducing our analysis in the paper titled: Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency

Comments

Morphological Analysis Examples

Releases(v1.0.0)

v1.0.0(Sep 24, 2021)

Owner

Pranaydeep Singh

Unsupervised Representation Learning via Neural Activation Coding

Code for Multiple Instance Active Learning for Object Detection, CVPR 2021

Calling Julia from Python - an experiment on data loading

NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling @ INTERSPEECH 2021 Accepted

Code for our CVPR 2021 Paper "Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes".

This is the code repository for the paper "Identification of the Generalized Condorcet Winner in Multi-dueling Bandits" (NeurIPS 2021).

Exe-to-xlsm - Simple script to create VBscript of exe and inject to xlsm

PINN Burgers - 1D Burgers equation simulated by PINN

Code for Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights

A Pytorch Implementation of Domain adaptation of object detector using scissor-like networks

ManipNet: Neural Manipulation Synthesis with a Hand-Object Spatial Representation - SIGGRAPH 2021

(ICCV 2021) Official code of "Dressing in Order: Recurrent Person Image Generation for Pose Transfer, Virtual Try-on and Outfit Editing."

A Python package for generating concise, high-quality summaries of a probability distribution

Nerf pl - NeRF (Neural Radiance Fields) and NeRF in the Wild using pytorch-lightning

Code for the paper "Attention Approximates Sparse Distributed Memory"

Playing around with FastAPI and streamlit to create a YoloV5 object detector

NEO: Non Equilibrium Sampling on the orbit of a deterministic transform

Probabilistic Programming and Statistical Inference in PyTorch

Unofficial PyTorch Implementation of "DOLG: Single-Stage Image Retrieval with Deep Orthogonal Fusion of Local and Global Features"

This reposityory contains the PyTorch implementation of our paper "Generative Dynamic Patch Attack".