Machine Learning Course Project, IMDB movie review sentiment analysis by lstm, cnn, and transformer

Overview

IMDB Sentiment Analysis

This is the final project of Machine Learning Courses in Huazhong University of Science and Technology, School of Artificial Intelligence and Automation

Training

To train a model (CNN, LSTM, Transformer), simply run

python train.py --cfg <./model/xxx> --save <./save/>

You can change the configuration in config.

Model

LSTM

we follow the origin LSTM as possible

lstm

CNN

we adopt the methods mentioned in Effective Use of Word Order for Text Categorization with Convolutional Neural Networks

cnn

Transformer

We use the original Transformer Encoder as Attention is all you need and use the concept of CLS Token as BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

trans

Experiment result

Model Comparison

Model Accuracy
LSTM 89.02
Transformer 87.47
CNN 88.66
Fine-tuned BERT 93.43

LSTM

Batch size
Batch size Loss Accuracy
64 0.4293 0.8802
128 0.4298 0.8818
256 0.4304 0.8836
512 0.4380 0.8807
Embedding Size
Embedding size train Loss train Accuracy val loss val accuracy
32 0.4021 0.9127 0.4419 0.8707
64 0.3848 0.9306 0.4297 0.8832
128 0.3772 0.9385 0.4265 0.8871
256 0.3584 0.9582 0.4303 0.8825
512 0.3504 0.9668 0.4295 0.8838
Drop out
Drop out rate Train Loss Train Accuracy Test loss Test Accuracy
0.0 0.3554 0.9623 0.4428 0.8704
0.1 0.3475 0.9696 0.4353 0.8780
0.2 0.3516 0.9652 0.4312 0.8825
0.3 0.3577 0.9589 0.4292 0.8844
0.4 0.3587 0.9576 0.4272 0.8868
0.5 0.3621 0.9544 0.4269 0.8865
0.6 0.3906 0.9242 0.4272 0.8863
0.7 0.3789 0.9356 0.4303 0.8826
0.8 0.3939 0.9204 0.4311 0.8826
0.9 0.4211 0.8918 0.4526 0.8584
Weight decay
Weight decay train loss train accuracy test loss test accuracy
1.0e-8 0.3716 0.9436 0.4261 0.8876
1.0e-7 0.3803 0.9349 0.4281 0.8862
1.0e-6 0.3701 0.9456 0.4264 0.8878
1.0e-5 0.3698 0.9461 0.4283 0.8850
1.0e-4 0.3785 0.9377 0.4318 0.8806
Number layers

Number of LSTM blocks

Number layers train loss train accuracy test loss test accuracy
1 0.3786 0.9364 0.4291 0.8844
2 0.3701 0.9456 0.4264 0.8878
3 0.3707 0.9451 0.4243 0.8902
4 0.3713 0.9446 0.4279 0.8857

CNN

out channel size
out size train acc test acc
8 0.9679 0.8743
16 0.9791 0.8767
32 0.9824 0.8811
64 0.9891 0.8848
128 0.9915 0.8824
256 0.9909 0.8827
512 0.9920 0.8841
1024 0.9959 0.8833
multi scale filter
Number train acc test acc
1 [5] 0.9698 0.8748
2 [5, 11] 0.9852 0.8827
3 [5, 11, 17] 0.9890 0.8850
4 [5, 11, 17, 23] 0.9915 0.8848
5 [5, 11, 17, 23, 29] 0.9924 0.8842
6 [5, 11, 17, 23, 29, 35] 0.9930 0.8836
step train acc test acc
2 [5 7 9] 0.9878 0.8816
4 [5 9 11] 0.9890 0.8816
6 [5 11 17] 0.9919 0.8834
8 [5 13 21] 0.9884 0.8836
10[5 15 25] 0.9919 0.8848
12[5 17 29] 0.9898 0.8812
14[5 29 43] 0.9935 0.8809
Owner
Daniel
Daniel
Spooky Skelly For Python

_____ _ _____ _ _ _ | __| ___ ___ ___ | |_ _ _ | __|| |_ ___ | || | _ _ |__ || . || . || . || '

Kur0R1uka 1 Dec 23, 2021
(ACL-IJCNLP 2021) Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models.

BERT Convolutions Code for the paper Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models. Contains expe

mlpc-ucsd 21 Jul 18, 2022
Journey is a NLP-Powered Developer assistant

Journey Journey is a NLP-Powered Developer assistant Using on the powerful Natural Language Processing library Mindmeld, this projects aims to assist

Christian Eilers 21 Dec 11, 2022
Galois is an auto code completer for code editors (or any text editor) based on OpenAI GPT-2.

Galois is an auto code completer for code editors (or any text editor) based on OpenAI GPT-2. It is trained (finetuned) on a curated list of approximately 45K Python (~470MB) files gathered from the

Galois Autocompleter 91 Sep 23, 2022
Finally, some decent sample sentences

tts-dataset-prompts This repository aims to be a decent set of sentences for people looking to clone their own voices (e.g. using Tacotron 2). Each se

hecko 19 Dec 13, 2022
Script and models for clustering LAION-400m CLIP embeddings.

clustering-laion400m Script and models for clustering LAION-400m CLIP embeddings. Models were fit on the first million or so image embeddings. A subje

Peter Baylies 22 Oct 04, 2022
RuCLIP tiny (Russian Contrastive Language–Image Pretraining) is a neural network trained to work with different pairs (images, texts).

RuCLIPtiny Zero-shot image classification model for Russian language RuCLIP tiny (Russian Contrastive Language–Image Pretraining) is a neural network

Shahmatov Arseniy 26 Sep 20, 2022
Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration

Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration This is the official repository for the EMNLP 2021 long pa

70 Dec 11, 2022
Shirt Bot is a discord bot which uses GPT-3 to generate text

SHIRT BOT · Shirt Bot is a discord bot which uses GPT-3 to generate text. Made by Cyclcrclicly#3420 (474183744685604865) on Discord. Support Server EX

31 Oct 31, 2022
Library for Russian imprecise rhymes generation

TOM RHYMER Library for Russian imprecise rhymes generation. Quick Start Generate rhymes by any given rhyme scheme (aabb, abab, aaccbb, etc ...): from

Alexey Karnachev 6 Oct 18, 2022
BERT Attention Analysis

BERT Attention Analysis This repository contains code for What Does BERT Look At? An Analysis of BERT's Attention. It includes code for getting attent

Kevin Clark 401 Dec 11, 2022
Lightweight utility tools for the detection of multiple spellings, meanings, and language-specific terminology in British and American English

Breame ( British English and American English) Breame is a lightweight Python package with a number of utility tools to aid in the detection of words

Charles 8 Oct 10, 2022
An easier way to build neural search on the cloud

An easier way to build neural search on the cloud Jina is a deep learning-powered search framework for building cross-/multi-modal search systems (e.g

Jina AI 17.1k Jan 09, 2023
Text-to-Speech for Belarusian language

title emoji colorFrom colorTo sdk app_file pinned Belarusian TTS 🐸 green green gradio app.py false Belarusian TTS 📢 🤖 Belarusian TTS (text-to-speec

Yurii Paniv 1 Nov 27, 2021
中文問句產生器;使用台達電閱讀理解資料集(DRCD)

Transformer QG on DRCD The inputs of the model refers to we integrate C and A into a new C' in the following form. C' = [c1, c2, ..., [HL], a1, ..., a

Philip 1 Oct 22, 2021
Coreference resolution for English, French, German and Polish, optimised for limited training data and easily extensible for further languages

Coreferee Author: Richard Paul Hudson, Explosion AI 1. Introduction 1.1 The basic idea 1.2 Getting started 1.2.1 English 1.2.2 French 1.2.3 German 1.2

Explosion 70 Dec 12, 2022
SGMC: Spectral Graph Matrix Completion

SGMC: Spectral Graph Matrix Completion Code for AAAI21 paper "Scalable and Explainable 1-Bit Matrix Completion via Graph Signal Learning". Data Format

Chao Chen 8 Dec 12, 2022
Adversarial Examples for Extreme Multilabel Text Classification

Adversarial Examples for Extreme Multilabel Text Classification The code is adapted from the source codes of BERT-ATTACK [1], APLC_XLNet [2], and Atte

1 May 14, 2022
Search with BERT vectors in Solr and Elasticsearch

Search with BERT vectors in Solr and Elasticsearch

Dmitry Kan 123 Dec 29, 2022