A Partition Filter Network for Joint Entity and Relation Extraction EMNLP 2021

Overview

PFN (Partition Filter Network)

This repository contains codes of the official implementation for the paper A Partition Filter Network for Joint Entity and Relation Extraction EMNLP 2021 [PDF] [PPT]

Quick links

Model Overview

In this work, we present a new framework equipped with a novel recurrent encoder named partition filter encoder designed for multi-task learning. The encoder enforces bilateral interaction between NER and RE in two ways:

  1. The shared partition represents inter-task information and is equally accessible to both tasks, allowing for balanced interaction between NER and RE.
  2. The task partitions represent intra-task information and are formed through concerted efforts of entity and relation gates, making sure that encoding process of entity and relation features are dependent upon each other.

Preparation

Environment Setup

The experiments were performed using one single NVIDIA-RTX3090 GPU. The dependency packages can be installed with the following command:

pip install -r requirements.txt

Also, make sure that the python version is 3.7.10

Data Acquisition and Preprocessing

This is the first work that covers all the mainstream English datasets for evaluation, including [NYT, WEBNLG, ADE, ACE2005, ACE2004, SCIERC, CONLL04]. Please follow the instructions of reademe.md in each dataset folder in ./data/ for data acquisition and preprocessing.

Custom Dataset

If your custom dataset has a large number of triples that contain head-overlap entities (common in Chinese dataset), accuracy of the orignal PFN will not be good.

The orignal one will not be able to decode triples with head-overlap entities. For example, if New York and New York City are both entities, and there exists a RE prediction such as (new, cityof, USA), we cannot know what New corresponds to.

Luckily, the impact on evaluation of English dataset is limited, since such triple is either filtered out (for ADE) or rare (one in test set of SciERC, one in ACE04, zero in other datasets).

You can use our updated PFN-nested to handle the issue. PFN-nested is an enhanced version of PFN. This model is better in leveraging entity tail information and capable of handling nested triple prediction. For usage, replace the files in the root directory with the files in the PFN-nested folder, then follow the directions in Quick Start.

Performance comparison in SciERC

Model NER RE
PFN 66.8 38.4
PFN-nested 67.9 38.7

Quick Start

Model Training

The training command-line is listed below (command for CONLL04 is in Evaluation on CoNLL04):

python main.py \
--data ${NYT/WEBNLG/ADE/ACE2005/ACE2004/SCIERC} \
--do_train \
--do_eval \
--embed_mode ${bert_cased/albert/scibert} \
--batch_size ${20 (for most datasets) /4 (for SCIERC)} \
--lr ${0.00002 (for most datasets) /0.00001 (for SCIERC)} \
--output_file ${the name of your output files, e.g. ace_test} \
--eval_metric ${micro/macro} 

After training, you will obtain three files in the ./save/${output_file}/ directory:

  • ${output_file}.log records the logging information.
  • ${output_file}.txt records loss, NER and RE results of dev set and test set for each epoch.
  • ${output_file}.pt is the saved model with best average F1 results of NER and RE in the dev set.

Evaluation on Pre-trained Model

The evaluation command-line is listed as follows:

python eval.py \
--data ${NYT/WEBNLG/ADE/ACE2005/ACE2004/SCIERC} \
--eval_metric ${micro/macro} \
--model_file ${the path of saved model you want to evaluate. e.g. save/ace_test.pt} \
--embed_mode ${bert_cased/albert/scibert}

Inference on Customized Input

If you want to evaluate the model with customized input, please run the following code:

python inference.py \
--model_file ${the path of your saved model} \
--sent ${sentence you want to evaluate, str type restricted}

{model_file} must contain information about the datasets the model trained on (web/nyt/ade/ace/sci) and the type of pretrained embedding the model uses (albert/bert/scibert). For example, model_file could be set as "web_bert.pt"

Example

input:
python inference.py \
--model_file save/sci_test_scibert.pt \
--sent "In this work , we present a new framework equipped with a novel recurrent encoder   
        named partition filter encoder designed for multi-task learning ."

result:
entity_name: framework, entity type: Generic
entity_name: recurrent encoder, entity type: Method
entity_name: partition filter encoder, entity type: Method
entity_name: multi-task learning, entity type: Task
triple: recurrent encoder, Used-for, framework
triple: recurrent encoder, Part-of, framework
triple: recurrent encoder, Used-for, multi-task learning
triple: partition filter encoder, Hyponym-of, recurrent encoder
triple: partition filter encoder, Used-for, multi-task learning



input:  
python inference.py \
--model_file save/ace_test_albert.pt \
--sent "As Williams was struggling to gain production and an audience for his work in the late 1930s ,  
        he worked at a string of menial jobs that included a stint as caretaker on a chicken ranch in   
        Laguna Beach , California . In 1939 , with the help of his agent Audrey Wood , Williams was 
        awarded a $1,000 grant from the Rockefeller Foundation in recognition of his play Battle of 
        Angels . It was produced in Boston in 1940 and was poorly received ."

result:
entity_name: Williams, entity type: PER
entity_name: audience, entity type: PER
entity_name: his, entity type: PER
entity_name: he, entity type: PER
entity_name: caretaker, entity type: PER
entity_name: ranch, entity type: FAC
entity_name: Laguna Beach, entity type: GPE
entity_name: California, entity type: GPE
entity_name: his, entity type: PER
entity_name: agent, entity type: PER
entity_name: Audrey Wood, entity type: PER
entity_name: Williams, entity type: PER
entity_name: Rockefeller Foundation, entity type: ORG
entity_name: his, entity type: PER
entity_name: Boston, entity type: GPE
triple: caretaker, PHYS, ranch
triple: ranch, PART-WHOLE, Laguna Beach
triple: Laguna Beach, PART-WHOLE, California

Evaluation on CoNLL04

We also run the test on the dataset CoNLL04, but we did not report the results in our paper due to several reasons:

The command for running CoNLL04 is listed below:

python main.py \
--data CONLL04 \
--do_train \
--do_eval \
--embed_mode albert \
--batch_size 10 \
--lr 0.00002 \
--output_file ${the name of your output files} \
--eval_metric micro \
--clip 1.0 \
--epoch 200

Pre-trained Models and Training Logs

We provide you with pre-trained models for NYT/WEBNLG/ACE2005/ACE2004/SCIERC/CONLL04, along with recorded results of each epoch, identical with training results under the specified configurations above.

Download Links

Due to limited space in google drive, 10-fold model files for ADE are not available to you (training record still available).

After downloading the linked files below, unzip them and put ${data}_test.pt in the directory of ./save/ before running eval.py. Also, ${data}_test.txt and ${data}_test.log records the results of each epoch. You should check that out as well.

Dataset File Size Embedding Download
NYT 393MB Bert-base-cased Link
WebNLG 393MB Bert-base-cased Link
ACE05 815MB Albert-xxlarge-v1 Link
ACE04 3.98GB Albert-xxlarge-v1 Link
SciERC 399MB Scibert-uncased Link
ADE 214KB Bert + Albert Link
CoNLL04 815MB Albert-xxlarge-v1 Link

Result Display

F1 results on NYT/WebNLG/ACE05/SciERC:

Dataset Embedding NER RE
NYT Bert-base-cased 95.8 92.4
WebNLG Bert-base-cased 98.0 93.6
ACE05 Albert-xxlarge-v1 89.0 66.8
SciERC Scibert-uncased 66.8 38.4

F1 results on ACE04:

5-fold 0 1 2 3 4 Average
Albert-NER 89.7 89.9 89.5 89.7 87.6 89.3
Albert-RE 65.5 61.4 63.4 61.5 60.7 62.5

F1 results on CoNLL04:

Model Embedding Micro-NER Micro-RE
Table-sequence Albert-xxlarge-v1 90.1 73.6
PFN Albert-xxlarge-v1 89.6 75.0

F1 results on ADE:

10-fold 0 1 2 3 4 5 6 7 8 9 Average
Bert-NER 89.6 92.3 90.3 88.9 88.8 90.2 90.1 88.5 88.0 88.9 89.6
Bert-RE 80.5 85.8 79.9 79.4 79.3 80.5 80.0 78.1 76.2 79.8 80.0
Albert-NER 91.4 92.9 91.9 91.5 90.7 91.6 91.9 89.9 90.6 90.7 91.3
Albert-RE 83.9 86.8 82.8 83.2 82.2 82.4 84.5 82.3 81.9 82.2 83.2

Robustness Against Input Perturbation

We use robustness test to evaluate our model under adverse circumstances. In this case, we use the domain transformation methods of NER from Textflint.

The test files can be found in the folder of ./robustness_data/. Our reported results are evaluated with the linked ACE2005-albert model above. For each test file, move it to ./data/ACE2005/ and rename it as test_triples.json, then run eval.py with the instructions above.

Citation

Please cite our paper if it's helpful to you in your research.

@misc{yan2021partition,
      title={A Partition Filter Network for Joint Entity and Relation Extraction}, 
      author={Zhiheng Yan and Chong Zhang and Jinlan Fu and Qi Zhang and Zhongyu Wei},
      year={2021},
      eprint={2108.12202},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
Owner
zhy
Knowledge Graph, Information Extraction, Interpretability of NLP System
zhy
Code for "FPS-Net: A convolutional fusion network for large-scale LiDAR point cloud segmentation".

FPS-Net Code for "FPS-Net: A convolutional fusion network for large-scale LiDAR point cloud segmentation", accepted by ISPRS journal of Photogrammetry

15 Nov 30, 2022
[ICML 2020] DrRepair: Learning to Repair Programs from Error Messages

DrRepair: Learning to Repair Programs from Error Messages This repo provides the source code & data of our paper: Graph-based, Self-Supervised Program

Michihiro Yasunaga 155 Jan 08, 2023
Official repository of OFA. Paper: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

Paper | Blog OFA is a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks (e.g., image gene

OFA Sys 1.4k Jan 08, 2023
Unofficial Pytorch Lightning implementation of Contrastive Syn-to-Real Generalization (ICLR, 2021)

Unofficial Pytorch Lightning implementation of Contrastive Syn-to-Real Generalization (ICLR, 2021)

Gyeongjae Choi 17 Sep 23, 2021
Pipeline code for Sequential-GAM(Genome Architecture Mapping).

Sequential-GAM Pipeline code for Sequential-GAM(Genome Architecture Mapping). mapping whole_preprocess.sh include the whole processing of mapping. usa

3 Nov 03, 2022
領域を指定し、キーを入力することで画像を保存するツールです。クラス分類用のデータセット作成を想定しています。

image-capture-class-annotation 領域を指定し、キーを入力することで画像を保存するツールです。 クラス分類用のデータセット作成を想定しています。 Requirement OpenCV 3.4.2 or later Usage 実行方法は以下です。 起動後はマウスクリック4

KazuhitoTakahashi 5 May 28, 2021
a morph transfer UGATIT for image translation.

Morph-UGATIT a morph transfer UGATIT for image translation. Introduction 中文技术文档 This is Pytorch implementation of UGATIT, paper "U-GAT-IT: Unsupervise

55 Nov 14, 2022
This is a project based on retinaface face detection, including ghostnet and mobilenetv3

English | 简体中文 RetinaFace in PyTorch Chinese detailed blog:https://zhuanlan.zhihu.com/p/379730820 Face recognition with masks is still robust---------

pogg 59 Dec 21, 2022
[ACM MM 2021] Joint Implicit Image Function for Guided Depth Super-Resolution

Joint Implicit Image Function for Guided Depth Super-Resolution This repository contains the code for: Joint Implicit Image Function for Guided Depth

hawkey 78 Dec 27, 2022
PyTorch implementation of the NIPS-17 paper "Poincaré Embeddings for Learning Hierarchical Representations"

Poincaré Embeddings for Learning Hierarchical Representations PyTorch implementation of Poincaré Embeddings for Learning Hierarchical Representations

Facebook Research 1.6k Dec 25, 2022
The official MegEngine implementation of the ICCV 2021 paper: GyroFlow: Gyroscope-Guided Unsupervised Optical Flow Learning

[ICCV 2021] GyroFlow: Gyroscope-Guided Unsupervised Optical Flow Learning This is the official implementation of our ICCV2021 paper GyroFlow. Our pres

MEGVII Research 36 Sep 07, 2022
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped

CSWin-Transformer This repo is the official implementation of "CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows". Th

Microsoft 409 Jan 06, 2023
Learn the Deep Learning for Computer Vision in three steps: theory from base to SotA, code in PyTorch, and space-repetition with Anki

DeepCourse: Deep Learning for Computer Vision arthurdouillard.com/deepcourse/ This is a course I'm giving to the French engineering school EPITA each

Arthur Douillard 113 Nov 29, 2022
An Straight Dilated Network with Wavelet for image Deblurring

SDWNet: A Straight Dilated Network with Wavelet Transformation for Image Deblurring(offical) 1. Introduction This repo is not only used for our paper(

FlyEgle 41 Jan 04, 2023
a pytorch implementation of auto-punctuation learned character by character

Learning Auto-Punctuation by Reading Engadget Articles Link to Other of my work 🌟 Deep Learning Notes: A collection of my notes going from basic mult

Ge Yang 137 Nov 09, 2022
PyTorch implementation of Wide Residual Networks with 1-bit weights by McDonnell (ICLR 2018)

1-bit Wide ResNet PyTorch implementation of training 1-bit Wide ResNets from this paper: Training wide residual networks for deployment using a single

Sergey Zagoruyko 122 Dec 07, 2022
AI创造营 :Metaverse启动机之重构现世,结合PaddlePaddle 和 Wechaty 创造自己的聊天机器人

paddle-wechaty-Zodiac AI创造营 :Metaverse启动机之重构现世,结合PaddlePaddle 和 Wechaty 创造自己的聊天机器人 12星座若穿越科幻剧,会拥有什么超能力呢?快来迎接你的专属超能力吧! 现在很多年轻人都喜欢看科幻剧,像是复仇者系列,里面有很多英雄、超

105 Dec 22, 2022
A Closer Look at Structured Pruning for Neural Network Compression

A Closer Look at Structured Pruning for Neural Network Compression Code used to reproduce experiments in https://arxiv.org/abs/1810.04622. To prune, w

Bayesian and Neural Systems Group 140 Dec 05, 2022
X-modaler is a versatile and high-performance codebase for cross-modal analytics.

X-modaler X-modaler is a versatile and high-performance codebase for cross-modal analytics. This codebase unifies comprehensive high-quality modules i

910 Dec 28, 2022
In Search of Probeable Generalization Measures

In Search of Probeable Generalization Measures Exciting News! In Search of Probeable Generalization Measures has been accepted to the International Co

Mahdi S. Hosseini 6 Sep 11, 2022