Minimalist BERT implementation assignment for CS11-747

Overview

minbert Assignment

by Zhengbao Jiang, Shuyan Zhou, and Ritam Dutt

This is an exercise in developing a minimalist version of BERT, part of Carnegie Mellon University's CS11-747: Neural Networks for NLP.

In this assignment, you will implement some important components of the BERT model to better understanding its architecture. You will then perform sentence classification on sst dataset and cfimdb dataset with the BERT model.

Assignment Details

Important Notes

  • Follow setup.sh to properly setup the environment and install dependencies.
  • There is a detailed description of the code structure in structure.md, including a description of which parts you will need to implement.
  • You are only allowed to use torch, no other external libraries are allowed (e.g., transformers).
  • We will run your code with the following commands, so make sure that whatever your best results are reproducible using these commands (where you replace ANDREWID with your andrew ID):
mkdir -p ANDREWID

python3 classifier.py --option [pretrain/finetune] --epochs NUM_EPOCHS --lr_pretrain LR_FOR_PRETRAINING --lr_finetune LR_FOR_FINETUNING --seed RANDOM_SEED

Reference accuracies:

Mean reference accuracies over 10 random seeds with their standard deviation shown in brackets.

Pretraining for SST: Dev Accuracy : 0.387 (0.008) Test Accuracy : 0.397 (0.013)

Finetuning for SST : Dev Accuracy : 0.520 (0.006) Test Accuracy : 0.525 (0.007)

Submission

The submission file should be a zip file with the following structure (assuming the andrew id is ANDREWID):

ANDREWID/
├── base_bert.py
├── bert.py
├── classifier.py
├── config.py
├── optimizer.py
├── sanity_check.py
├── tokenizer.py
├── utils.py
├── README.md
├── structure.md
├── sanity_check.data
├── sst-dev-output.txt 
├── sst-test-output.txt 
├── cfimdb-dev-output.txt 
├── cfimdb-test-output.txt 
└── setup.py

Grading

  • A+: You additionally implement something else on top of the requirements for A, and achieve significant accuracy improvements:
  • A: You implement all the missing pieces and the original classifier.py with --option finetune code that achieves comparable accuracy to our reference implementation
  • A-: You implement all the missing pieces and the original classifier.py with --option pretrain code that achieves comparable accuracy to our reference implementation
  • B+: All missing pieces are implemented and pass tests in sanity_check.py, but accuracy is not comparable to the reference.
  • B or below: Some parts of the missing pieces are not implemented.

Acknowledgement

Parts of the code are from the transformers library (Apache License 2.0).

Owner
Graham Neubig
Graham Neubig
Utility to play with ADCS, allows to request tickets and collect information about related objects

certi Utility to play with ADCS, allows to request tickets and collect information about related objects. Basically, it's the impacket copy of Certify

Eloy 185 Dec 29, 2022
This repo is a collection of programs and websites templates too

📢 Register here for Hacktoberfest and make four pull requests (PRs) between October 1st-31st to grab free SWAGS 🔥 . IMPORTANT While making pull requ

Binayak Jha - 2 7 Oct 03, 2022
This module is for finding the execution time of a whole python program

exetime 3.8 This module is for finding the execution time of a whole program How to install $ pip install exetime Contents: General Information Instru

Saikat Das 4 Oct 18, 2021
Recreating my first CRUD in python, but now more professional

Recreating my first CRUD in python, but now more professional

Ricardo Deo Sipione Augusto 2 Nov 27, 2021
COVID-19 case tracker in Dash

covid_dashy_personal This is a personal project to build a simple COVID-19 tracker for Australia with Dash. Key functions of this dashy will be to Dis

Jansen Zhang 1 Nov 30, 2021
Sheet2export - FreeCAD macro to export spreadsheet

Description This is FreeCAD macro to export spreadsheet to file.

Darek L 3 Jul 09, 2022
✨ Udemy Coupon Finder For Discord. Supports Turkish & English Language.

Udemy Course Finder Bot | Udemy Kupon Bulucu Botu This bot finds new udemy coupons and sends to the channel. Before Setup You must have python = 3.6

Penguen 4 May 04, 2022
Practice10 - Operasi String With Python

Operasi String MY SOSIAL MEDIA : Apa itu Python String ? String adalah urutan si

Maulana Reza Badrudin 1 Jan 05, 2022
You will need to install a few python packages for this one.

Features Bait support Auto repair will repair every 10 catches Anti detection (still a work in progress) but using random times and click positions Pr

12 Sep 21, 2022
My solutions to Advent of Code 2021 (written in Python)

Advent of Code 2021 This repository contains my solutions for the 2021 edition of Advent of Code. Please do not expect perfectly polished solutions, m

Nils 2 May 29, 2022
Simple calculator with random number button and dark gray theme created with PyQt6

Calculator Application Simple calculator with random number button and dark gray theme created with : PyQt6 Python 3.9.7 you can download the dark gra

Flamingo 2 Mar 07, 2022
GibMacOS - Py2/py3 script that can download macOS components direct from Apple

Py2/py3 script that can download macOS components direct from Apple Can also now build Internet Recovery USB installers from Windows using dd and 7zip

CorpNewt 4.8k Jan 02, 2023
Donatus Prince 6 Feb 25, 2022
A Python application that helps users determine their calorie intake, and automatically generates customized weekly meal and workout plans based on metrics computed using their physical parameters

A Python application that helps users determine their calorie intake, and automatically generates customized weekly meal and workout plans based on metrics computed using their physical parameters

Anam Iqbal 1 Jan 13, 2022
JPMC Virtual Experience

This repository contains the submitted patch files along with raw files of the various tasks assigned by JPMorgan Chase & Co. through its Software Engineering Virtual Experience Program on Forage (fo

Vardhini K 1 Dec 05, 2021
A python implementation of differentiable quality diversity.

Differentiable Quality Diversity This repository is the official implementation of Differentiable Quality Diversity.

ICAROS 41 Nov 30, 2022
Notes on the Deep Learning book from Ian Goodfellow, Yoshua Bengio and Aaron Courville (2016)

The Deep Learning Book - Goodfellow, I., Bengio, Y., and Courville, A. (2016) This content is part of a series following the chapter 2 on linear algeb

hadrienj 1.7k Jan 07, 2023
Grouping nucleotide coordinate ranges.

NuclRanger Grouping nucleotide coordinate ranges. A quick pre-processing step for "bedtools getfasta":- https://bedtools.readthedocs.io/en/latest/cont

Sujanavan Tiruvayipati 1 Oct 04, 2022
Diff Match Patch is a high-performance library in multiple languages that manipulates plain text.

The Diff Match and Patch libraries offer robust algorithms to perform the operations required for synchronizing plain text. Diff: Compare two blocks o

Google 5.9k Dec 30, 2022
A totally unrealistic cell growth/reproduction simulation.

A totally unrealistic cell growth/reproduction simulation.

Andrien Wiandyano 1 Oct 24, 2021