p-tuning_NLU

Overview

这个小项目是受乐于分享的苏剑林大佬这篇p-tuning 文章启发，也实现了个使用P-tuning进行NLU分类的任务，思路是一样的，prompt实现方式有不同，这里是将[unused*]的embeddings参数抽取出用于初始化prompt_embed后，再接一个lstm和mlp用于关联各prompt，与最初p-tuning提出《GPT Understands, Too》的实现一样，结果显示在few-shot上p-tuning非常接近finetune效果。

Dataset

数据是情感分类，下载地址百度网盘提取码：osja

Evaluation

1. finetune

python few_shot_finetune.py

测试集效果：

epoch: 0 - acc: 0.897679 - best_test_acc: 0.8976788252013264
epoch: 1 - acc: 0.876362 - best_test_acc: 0.8976788252013264
epoch: 2 - acc: 0.884889 - best_test_acc: 0.8976788252013264
epoch: 3 - acc: 0.884415 - best_test_acc: 0.8976788252013264
epoch: 4 - acc: 0.884415 - best_test_acc: 0.8976788252013264

全量参数对小样本进行finetune，仅1个epoch就收敛了

2. p-tuning

python few_shot_ptuning.py

测试集效果：

epoch: 0 - acc: 0.546660 - best_test_acc: 0.5466603505447655
epoch: 1 - acc: 0.687826 - best_test_acc: 0.6878256750355282
epoch: 2 - acc: 0.737091 - best_test_acc: 0.7370914258645191
epoch: 3 - acc: 0.722406 - best_test_acc: 0.7370914258645191
epoch: 4 - acc: 0.776883 - best_test_acc: 0.7768829938417812
epoch: 5 - acc: 0.805306 - best_test_acc: 0.8053055423969683
epoch: 6 - acc: 0.833254 - best_test_acc: 0.8332543818095689
epoch: 7 - acc: 0.837991 - best_test_acc: 0.8379914732354334
epoch: 8 - acc: 0.854571 - best_test_acc: 0.8545712932259593
epoch: 9 - acc: 0.858361 - best_test_acc: 0.8583609663666508
epoch: 10 - acc: 0.856466 - best_test_acc: 0.8583609663666508
epoch: 11 - acc: 0.853150 - best_test_acc: 0.8583609663666508
epoch: 12 - acc: 0.868783 - best_test_acc: 0.8687825675035529
epoch: 13 - acc: 0.877309 - best_test_acc: 0.877309332070109
epoch: 14 - acc: 0.873993 - best_test_acc: 0.877309332070109
epoch: 15 - acc: 0.877783 - best_test_acc: 0.8777830412126955
epoch: 16 - acc: 0.882994 - best_test_acc: 0.8829938417811464
epoch: 17 - acc: 0.881573 - best_test_acc: 0.8829938417811464
epoch: 18 - acc: 0.889626 - best_test_acc: 0.8896257697773567
epoch: 19 - acc: 0.877783 - best_test_acc: 0.8896257697773567

仅prompt_embed和lstm及mlp去做p-tuning，20个epoch后接近收敛，acc=0.8896，略小于finetun的acc 0.8977

附上苏神结果对比：

p-tuning for few-shot NLU task

Related tags

Overview

p-tuning_NLU

Overview

Dataset

Evaluation

Owner

Opal-lang - A WIP programming language based on Python

German Text-To-Speech Engine using Tacotron and Griffin-Lim

Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields

Build Text Rerankers with Deep Language Models

A fast, efficient universal vector embedding utility package.

MMDA - multimodal document analysis

मराठी भाषा वाचविण्याचा एक प्रयास. इंग्रजी ते मराठीचा शब्दकोश. An attempt to preserve the Marathi language. A lightweight and ad free English to Marathi thesaurus.

Chinese version of GPT2 training code, using BERT tokenizer.

Implementation of paper Does syntax matter? A strong baseline for Aspect-based Sentiment Analysis with RoBERTa.

Google and Stanford University released a new pre-trained model called ELECTRA

小布助手对话短文本语义匹配的一个baseline

Pretty-doc - Composable text objects with python

Materials (slides, code, assignments) for the NYU class I teach on NLP and ML Systems (Master of Engineering).

Text classification on IMDB dataset using Keras and Bi-LSTM network

In this project, we compared Spanish BERT and Multilingual BERT in the Sentiment Analysis task.

Shirt Bot is a discord bot which uses GPT-3 to generate text

In this project, we aim to achieve the task of predicting emojis from tweets. We aim to investigate the relationship between words and emojis.

Graph Coloring - Weighted Vertex Coloring Problem

Code for the project carried out fulfilling the course requirements for Fall 2021 NLP at NYU

Transformer - A TensorFlow Implementation of the Transformer: Attention Is All You Need