Minimalist BERT implementation assignment for CS11-747

Overview

minbert Assignment

by Zhengbao Jiang, Shuyan Zhou, and Ritam Dutt

This is an exercise in developing a minimalist version of BERT, part of Carnegie Mellon University's CS11-747: Neural Networks for NLP.

In this assignment, you will implement some important components of the BERT model to better understanding its architecture. You will then perform sentence classification on sst dataset and cfimdb dataset with the BERT model.

Assignment Details

Important Notes

  • Follow setup.sh to properly setup the environment and install dependencies.
  • There is a detailed description of the code structure in structure.md, including a description of which parts you will need to implement.
  • You are only allowed to use torch, no other external libraries are allowed (e.g., transformers).
  • We will run your code with the following commands, so make sure that whatever your best results are reproducible using these commands (where you replace ANDREWID with your andrew ID):
mkdir -p ANDREWID

python3 classifier.py --option [pretrain/finetune] --epochs NUM_EPOCHS --lr_pretrain LR_FOR_PRETRAINING --lr_finetune LR_FOR_FINETUNING --seed RANDOM_SEED

Reference accuracies:

Mean reference accuracies over 10 random seeds with their standard deviation shown in brackets.

Pretraining for SST: Dev Accuracy : 0.387 (0.008) Test Accuracy : 0.397 (0.013)

Finetuning for SST : Dev Accuracy : 0.520 (0.006) Test Accuracy : 0.525 (0.007)

Submission

The submission file should be a zip file with the following structure (assuming the andrew id is ANDREWID):

ANDREWID/
├── base_bert.py
├── bert.py
├── classifier.py
├── config.py
├── optimizer.py
├── sanity_check.py
├── tokenizer.py
├── utils.py
├── README.md
├── structure.md
├── sanity_check.data
├── sst-dev-output.txt 
├── sst-test-output.txt 
├── cfimdb-dev-output.txt 
├── cfimdb-test-output.txt 
└── setup.py

Grading

  • A+: You additionally implement something else on top of the requirements for A, and achieve significant accuracy improvements:
  • A: You implement all the missing pieces and the original classifier.py with --option finetune code that achieves comparable accuracy to our reference implementation
  • A-: You implement all the missing pieces and the original classifier.py with --option pretrain code that achieves comparable accuracy to our reference implementation
  • B+: All missing pieces are implemented and pass tests in sanity_check.py, but accuracy is not comparable to the reference.
  • B or below: Some parts of the missing pieces are not implemented.

Acknowledgement

Parts of the code are from the transformers library (Apache License 2.0).

Owner
Graham Neubig
Graham Neubig
Flow control is the order in which statements or blocks of code are executed at runtime based on a condition. Learn Conditional statements, Iterative statements, and Transfer statements

03_Python_Flow_Control Introduction 👋 The control flow statements are an essential part of the Python programming language. A control flow statement

Milaan Parmar / Милан пармар / _米兰 帕尔马 209 Oct 31, 2022
An end-to-end encrypted chat

An end-to-end encrypted chat, that allows users to anonymously talk without ip logs, personal info, or need for registration.

Privalise 1 Nov 27, 2021
Huggingface package for the discrete VAE used for DALL-E.

DALL-E-Tokenizer Huggingface package for the discrete VAE used for DALL-E.

MyungHoon Jin 5 Sep 01, 2021
Just some mtk tool for exploitation, reading/writing flash and doing crazy stuff

Just some mtk tool for exploitation, reading/writing flash and doing crazy stuff. For linux, a patched kernel is needed (see Setup folder) (except for read/write flash). For windows, you need to inst

Bjoern Kerler 1.1k Dec 31, 2022
Localization and multifractal properties of the long-range Kitaev chain in the presence of an Aubry-André-Harper modulation

This repository contains the code for the paper Localization and multifractal properties of the long-range Kitaev chain in the presence of an Aubry-André-Harper modulation.

Joana Fraxanet 2 Apr 17, 2022
Type Persian without confusing words for yourself and others, in Adobe Connect

About In the Adobe Connect chat section, to type in Persian or Arabic, the written words will be confused and will be written and sent illegibly (This

Matin Najafi 23 Nov 26, 2021
A basic python project which replicates the functionalities on an 8 Ball.

Magic-8-Ball To the people who wish to make decisions using a Magic 8 Ball but can't get one? I gotchu. This is a basic python project which replicate

3 Jun 24, 2021
My Solutions to 120 commonly asked data science interview questions.

Data_Science_Interview_Questions Introduction 👋 Here are the answers to 120 Data Science Interview Questions The above answer some is modified based

Milaan Parmar / Милан пармар / _米兰 帕尔马 181 Dec 31, 2022
This Python3 script will monitor Upwork RSS feed and then email you the results.

Upwork RSS Parser This Python3 script will monitor Upwork RSS feed and then email you the results. Table of Contents General Info Technologies Used Fe

Chris 5 Nov 29, 2021
msImpersonate - User account impersonation written in pure Python3

msImpersonate v1.0 msImpersonate is a Python-native user impersonation tool that is capable of impersonating local or network user accounts with valid

Joe Helle 90 Dec 16, 2022
MDAnalysis tool to calculate membrane curvature.

The MDAkit for membrane curvature analysis is part of the Google Summer of Code program and it is linked to a Code of Conduct.

MDAnalysis 19 Oct 20, 2022
The Zig programming language, packaged for PyPI

Zig PyPI distribution This repository contains the script used to repackage the releases of the Zig programming language as Python binary wheels. This

Zig Programming Language 100 Nov 04, 2022
Different steganography methods with examples and my own small image database

literally-the-most-useless-project [Different steganography methods with examples and my own small image database] This project currently contains thr

Kamyishka 1 Dec 09, 2022
Time python - Códigos para auxiliar e mostrar formas de como fazer um relógio e manipular o seu tempo

Time_python Códigos para auxiliar e mostrar formas de como fazer um relógio e manipular o seu tempo. Bibliotecas Nestes foram usadas bibliotecas nativ

Eduardo Henrique 1 Jan 03, 2022
Battery conservation Python script for ubuntu to enable battery conservation mode at 60% 80% or 90%

Description Batteryconservation is a small python script wich creates an appindicator for ubuntu which can be used to enable / disable battery conserv

3 Jan 04, 2022
solsim is the Solana complex systems simulator. It simulates behavior of dynamical systems—DeFi protocols, DAO governance, cryptocurrencies, and more—built on the Solana blockchain

solsim is the Solana complex systems simulator. It simulates behavior of dynamical systems—DeFi protocols, DAO governance, cryptocurrencies, and more—built on the Solana blockchain

William Wolf 12 Jul 13, 2022
tg-nearby Trilateration of nearby Telegram users as described in my corresponding article.

tg-nearby Trilateration of nearby Telegram users as described in my corresponding article. Setup If you want to toy with the code in this repository

Maximilian Jugl 75 Dec 26, 2022
The LiberaPay archive module for the SeanPM life archive project.

By: Top README.md Read this article in a different language Sorted by: A-Z Sorting options unavailable ( af Afrikaans Afrikaans | sq Shqiptare Albania

Sean P. Myrick V19.1.7.2 1 Aug 26, 2022
AKSWINPOSTINIT -- AKS Windows node post provisioning initialization

AKSWINPOSTINIT -- AKS Windows node post provisioning initialization Features This is a tool that provides one-time powershell script initilization for

Ping He 3 Nov 25, 2021
To check my COVID-19 vaccine appointment, I wrote an infinite loop that sends me a Whatsapp message hourly using Twilio and Selenium. It works on my Raspberry Pi computer.

COVID-19_vaccine_appointment To check my COVID-19 vaccine appointment, I wrote an infinite loop that sends me a Whatsapp message hourly using Twilio a

Ayyuce Demirbas 24 Dec 17, 2022