Official repository for "PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation"

Last update: Oct 13, 2022

Related tags

Deep Learning pair-emnlp2020

Overview

pair-emnlp2020

Official repository for the paper:

Xinyu Hua and Lu Wang: PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation

If you find our work useful, please cite:

@inproceedings{hua-wang-2020-pair,
    title = "PAIR: Planning and Iterative Refinement in Pre-trained Transformersfor Long Text Generation",
    author = "Hua, Xinyu  and
      Wang, Lu",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
}

Requirements

Python 3.7
PyTorch 1.4.0
PyTorchLightning 0.9.0
transformers 3.3.0
numpy
tqdm
pycorenlp (for preprocessing nytimes data)
nltk (for preprocessing nytimes data)

Data

We release the data sets in the following link(1.2G uncompressed) Please download and uncompress the file, and put under ./data directory. For opinion and news domains, the The New York Times Annotated Corpus is licensed by LDC. We therefore only provide the ids for train/dev/test. Please follow the instructions to generate the dataset.

Text Planning

To train a BERT planner:

cd planning
python train.py \
    --data-path=../data/ \
    --domain=[arggen,opinion,news] \
    --exp-name=demo \
    --save-interval=1 \ # how frequent to save checkpoints 
    --max-epoch=30 \
    --lr=5e-4 \
    --warmup-updates=5000 \
    --train-set=train \
    --valid-set=dev \
    --tensorboard-logdir=tboard/ \
    --predict-keyphrase-offset \
    --max-samples=32 \ # max number of samples per batch
    [--quiet] \ # whether to print intermediate information

The checkpoints will be dumped to checkpoints/planning/[domain]/[exp-name]. Tensorboard will be available under planning/tboard/.

To run inference using a trained model, with greedy decoding:

cd planning
python decode.py \
    --data-path=../data/ \
    --domain=arggen \
    --test-set=test \
    --max-samples=32 \
    --predict-keyphrase-offset \
    --exp-name=demo \
    [--quiet]

The results will be saved to planning/output/.

Iterative Refinement

We provide implementations for four different setups:

Seq2seq: prompt -> tgt
KPSeq2seq: prompt + kp-set -> tgt
PAIR-light: prompt + kp-plan + masks -> tgt
PAIR-full: prompt + kp-plan + template -> tgt

To train a model:

cd refinement
python train.py \
    --domain=[arggen,opinion,news] \
    --setup=[seq2seq,kpseq2seq,pair-light,pair-full] \
    --train-set=train \
    --valid-set=dev \
    --train-batch-size=10 \
    --valid-batch-size=5 \
    --num-train-epochs=20 \
    --ckpt-dir=../checkpoints/[domain]/[setup]/demo \
    --tensorboard-dir=demo \
    [--quiet]

To run iterative refinement:

cd refinement
python generate.py \
    --domain=[arggen,opinion,news] \
    --setup=[seq2seq,kpseq2seq,pair-light,pair-full] \
    --test-set=test \
    --output-name=test_demo \
    --enforce-template-strategy=flexible \
    --do-sampling \
    --sampling-topk=100 \
    --sampling-topp=0.9 \
    --sample-times=3 \
    --ckpt-dir=../checkpoints/[domain]/[setup]/demo

Contact

Xinyu Hua (hua.x [at] northeastern.edu)

License

See the LICENSE file for details.

Official repository for "PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation"

Related tags

Overview

pair-emnlp2020

Requirements

Data

Text Planning

Iterative Refinement

Contact

License

Owner

Xinyu Hua

QueryInst: Parallelly Supervised Mask Query for Instance Segmentation

Illuminated3D This project participates in the Nasa Space Apps Challenge 2021.

Implementation of CVPR'21: RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction

The official PyTorch implementation for NCSNv2 (NeurIPS 2020)

A trashy useless Latin programming language written in python.

Fully-automated scripts for collecting AI-related papers

PyTorch 1.0 inference in C++ on Windows10 platforms

Deep Learning for 3D Point Clouds: A Survey (IEEE TPAMI, 2020)

Adversarial examples to the new ConvNeXt architecture

General neural ODE and DAE modules for power system dynamic modeling.

Finetune the base 64 px GLIDE-text2im model from OpenAI on your own image-text dataset

Stacked Hourglass Network with a Multi-level Attention Mechanism: Where to Look for Intervertebral Disc Labeling

nanodet_plus,yolov5_v6.0

Embeddinghub is a database built for machine learning embeddings.

An self sufficient AI that crawls the web to learn how to generate art from keywords

Genetic Algorithm, Particle Swarm Optimization, Simulated Annealing, Ant Colony Optimization Algorithm,Immune Algorithm, Artificial Fish Swarm Algorithm, Differential Evolution and TSP(Traveling salesman)

Pytorch implementation of "A simple neural network module for relational reasoning" (Relational Networks)

Unofficial Implementation of Oboe (SIGCOMM'18').

RealTime Emotion Recognizer for Machine Learning Study Jam's demo

Strongly local p-norm-cut algorithms for semi-supervised learning and local graph clustering