Benchmark for evaluating open-ended generation

Overview

OpenMEVA

Contributed by Jian Guan, Zhexin Zhang. Thank Jiaxin Wen for DeBugging.

OpenMEVA is a benchmark for evaluating open-ended story generation metrics (Please refer to the Paper List for more information about Open-eNded Language Generation tasks) described in the paper: OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics (ACL 2021 Long Paper). Besides, OpenMEVA also provides an open-source and extensible toolkit for metric implementation, evaluation, comparison, and analysis, as well as data perturbation techniques to help generate large numbers of customized test cases. We expect the toolkit to empower fast development of automatic metrics.

Contents

Introduction for Language Generation Evaluation

Since human evaluation is time-consuming, expensive, and difficult to reproduce, the community commonly uses automatic metrics for evaluation. We roughly divide existing metrics as follows:

  • Previous studies in conditional language generation tasks (e.g., machine translation) have developed several successful referenced metrics, which roughly quantify the lexical overlap (e.g., BLEU) or semantic entailment (e.g., BertScore) between a generated sample and the reference.
  • Referenced metrics correlate poorly with human judgments when evaluating open-ended language generation. Specifically, a generated sample can be reasonable if it is coherent to the given input, and self-consistent within its own context but not necessarily being similar to the reference in literal or semantics. To address the one-to-many issue, unreferenced metrics (e.g., UNION) are proposed to measure the quality of a generated sample without any reference.
  • Besides, some researchers propose to combine referenced and unreferenced metrics, i.e. hybrid metrics, which usually average two individual metric scores (e.g. RUBER) or learn from human preference (e.g., ADEM). However, ADEM is reported to lack generalization and robustness with limited human annotation.

The existing generation models are still far from human ability to generate reasonable texts, particularly for open-ended language generation tasks such as story generation. One important factor that hinders the research is the lack of powerful metrics for measuring generation quality. Therefore, we propose OpenMEVA as the standard paradigm for measuring progress of metrics.

Install

Clone the repository from our github page (don't forget to star us!)

git clone https://github.com/thu-coai/OpenMEVA.git

Then install all the requirements:

pip install -r requirements.txt

Then install the package with

python setup.py install

If you also want to modify the code, run this:

python setup.py develop

Toolkit

I. Metrics Interface

1. Metric List

We publish the standard implementation for the following metrics:

2. Usage

It is handy to construct a metric object and use it to evaluate given examples:

from eva.bleu import BLEU
metric = BLEU()

# for more information about the metric
print(metric.info)

# data is a list of dictionary [{"context": ..., "candidate":..., "reference": ...}]
print(metric.compute(data))

We present a python file test.py as an instruction to access the API.

These metrics are not exhaustive, it is a starting point for further metric research. We welcome any pull request for other metrics (requiring implementation of only three methods including __init__, info, compute).

3. Training Learnable Metrics

Execute the following command for training learnable metrics:

cd ./eva/model

# training language model for computing forward perplexity
bash ./run_language_modeling.sh

# training the unreferenced model for computing RUBER (RNN version)
bash ./run_ruber_unrefer.sh

# training the unreferenced model for computing RUBER (BERT version)
bash ./run_ruber_unrefer_bert.sh

# training the model for computing UNION
bash ./run_union.sh

II. Evaluating Human Scores

The python file test.py also includes detailed instruction to access the API for evaluating human scores.

1. Constructing

from eva.heva import Heva

# list of all possible human scores (int/float/str).
all_possible_score_list = [1,2,3,4,5]

# construct an object for following evaluation
heva = Heva(all_possible_score_list)

2. Consistency of human scores

# list of human score list, each row includes all the human scores for an example
human_score_list = [[1,3,2], [1,3,3], [2,3,1], ...]

print(heva.consistency(human_score_list))
# {"Fleiss's kappa": ..., "ICC correlation": ..., "Kendall-w":..., "krippendorff's alpha":...}
# the results includes correlation and p-value for significance test.

3. Mean Test for scores of examples from different source

# list of metric scores (float)
metric_score_1, metric_score_2 = [3.2, 2.4, 3.1,...], [3.5, 1.2, 2.3, ...]

# T-test for the means of two independent samples of scores.
print(heva.mean_test(metric_score_1, metric_score_2))
# {"t-statistic": ..., "p-value": ...}

4. Distribution of human scores

# list of human scores (float)
human_score = [2.0, 4.2, 1.2, 4.9, 2.6, 3.1, 4.0, 1.5,...]

# path for saving the figure of distribution
figure_path = "./figure"

# indicating the source of the annotated examples. default: ""
model_name = "gpt"

# plot the figure of distribution of human scores
heva.save_distribution_figure(score=human_score, save_path=figure_path, model_name=model_name, ymin=0, ymax=50)

5. Correlation between human and metric scores

# list of human scores (float)
human_score = [2.0, 4.2, 1.2, 4.9, 2.6, 3.1, 4.0, 1.5,...]

# list of metric scores (float)
metric_score = [3.2, 2.4, 3.1, 3.5, 1.2, 2.3, 3.5, 1.1,...]

# computing correlation
print(heva.correlation(metric_score, human_score))

# path for saving the figure of distribution
figure_path = "./figure"

# indicating the source of the metric scores. default: ""
metric_name = "bleu"

# plot the figure of metric score vs. human scores
heva.save_correlation_figure(human_score, metric_score, save_path=figure_path, metric_name=metric_name)

III. Perturbation Techniques

1. Perturbation List

We provide perturbation techniques in following aspects to create large scale test cases for evaluating comprehensive capabilities of metrics:

  • Lexical repetition

    • Repeating n-grams or sentences:

      He stepped on the stage and stepped on the stage.
  • Semantic repetition:

    • Repeating sentences with paraphrases by back translation:

      He has been from Chicago to Florida. He moved to Florida from Chicago.

  • Character behavior:

    • Reordering the subject and object of a sentence:

      Lars looked at the girl with desire.→ the girl looked at Lars with desire.
    • Substituting the personal pronouns referring to other characters:

      her mother took them to ... → their mother took her to ...
  • Common sense:

    • Substituting the head or tail entities in a commonsense triple of ConcepNet:

      Martha puts her dinner into theoven. She lays down fora quick nap. She oversleeps and runs into the kitchen (→ garden) to take out her burnt dinne.
  • Consistency:

    • Inserting or Deleting negated words or prefixes:

      She had (→ did not have) money to get vaccinated. She had a flu shot ...
      She agreed (→ disagreed) to get vaccinated.
    • Substituting words with antonyms:

      She is happy (→ upset) that she had a great time ...
  • Coherence:

    • Substituting words, phrases or sentences:

      Christmas was very soon. Kelly wanted to put up the Christmas tree. (→ Eventually it went into remission.)
  • Causal Relationship:

    • Reordering the cause and effect:

      the sky was clear so he could see clearly the boat. → he could see clearly the boat so the sky was clear.
    • Substituting the causality-related words randomly:

      the sky was clear so (→ because) he could see clearly the boat.
  • Temporal Relationship:

    • Reordering two sequential events:

      I eat one bite. Then I was no longer hungry.I was no longer hungry. Then I eat one bite.
    • Substituting the time-related words:

      After (→ Before) eating one bite I was no longer hungry.
  • Synonym:

    • Substituting a word with its synonym:

      I just purchased (→ bought) my uniforms.
  • Paraphrase:

    • Substituting a sentence with its paraphrase by back translation:

      Her dog doesn't shiver anymore.Her dog stops shaking.
  • Punctuation:

    • Inserting or Deleting inessential punctuation mark:

      Eventually,Eventually he became very hungry.
  • Contraction:

    • Contracting or Expanding contraction:

      I’ll (→ I will) have to keep waiting .
  • Typo:

    • Swapping two adjacent characters:

      that orange (→ ornage) broke her nose.
    • Repeating or Deleting a character:

      that orange (→ orannge) broke her nose.

2. Usage

It is handy to construct a perturbation object and use it to perturb given examples:

from eva.perturb.perturb import *
custom_name = "story"
method = add_typos(custom_name)

# data is a list of dictionary [{"id":0, "ipt": ..., "truth":...}, ...]
print(method.construct(data))
# the perturbed examples can be found under the directory "custom_name"

We present a python file test_perturb.py as an instruction to access the API.

You can download dependent files for some perturbation techniques by executing the following command:

cd ./eva/perturb
bash ./download.sh

You can also download them by THUCloud or Google Drive.

These perturbation techniques are not exhaustive, it is a starting point for further evaluation research. We welcome any pull request for other perturbation techniques (requiring implementation of only two methods including __init__, construct).

Note 📑 We adopt uda for back translation. We provide an example eva/perturb/back_trans_data/story_bt.json to indicate the format of the back translation result. And you can download the results for ROCStories and WritingPrompts by THUCloud or Google Drive.

Benchmark

I. Datasets

1. Machine-Generated Stories (MAGS) with manual annotation

We provide annotated stories from ROCStories (ROC) and WritingPrompts (WP). Some statistics are as follows:

Boxplot of annotation scores for each story source (Left: ROC, Right: WP):

2. Auto-Constructed Stories (ACTS)

We create large-scale test examples based on ROC and WP by aforementioned perturbation techniques. ACTS includes examples for different test types, i.e., discrimination test and invariance test.

  • The discrimination test requires metrics to distinguish human-written positive examples from negative ones. Wecreate each negative example by applying pertur-bation within an individual aspect. Besides, we also select different positive examples targeted for corresponding aspects. Below table shows the numbers of positive and negative examples in different aspects.

  • The invariance test expect the metric judgments to remain the same when we apply rationality-preserving perturbations, which means almost no influence on the quality of examples. The original examples can be either the human-written stories or the negative examples created in the discrimination test. Below table shows the numbers of original (also perturbed) positive and negative examples in different aspects.

3. Download & Data Instruction

You can download the whole dataset by THUCloud or Google Drive.

├── data
   └── `mags_data`
       ├── `mags_roc.json`	# sampled stories and corresponding human annotation.   
       ├── `mags_wp.json`		# sampled stories and corresponding human annotation.       
   └── `acts_data`
       ├── `roc`
              └── `roc_train_ipt.txt`	# input for training set
              └── `roc_train_opt.txt`	# output for training set
              └── `roc_valid_ipt.txt`	# input for validation set
              └── `roc_valid_opt.txt`	# output for validation set
              └── `roc_test_ipt.txt`	# input for test set
              └── `roc_test_opt.txt`	# output for test set
              └── `discrimination_test`                        
                 ├── `roc_lexical_rept.txt`
                 ├── `roc_lexical_rept_perturb.txt`										
                 ├── `roc_semantic_rept.txt`
                 ├── `roc_semantic_rept_perturb.txt`
                 ├── `roc_character.txt`
                 ├── `roc_character_perturb.txt`
                 ├── `roc_commonsense.txt`
                 ├── `roc_commonsense_perturb.txt`												
                 ├── `roc_coherence.txt`
                 ├── `roc_coherence_perturb.txt`
                 ├── `roc_consistency.txt`
                 ├── `roc_consistency_perturb.txt`								
                 ├── `roc_cause.txt`
                 ├── `roc_cause_perturb.txt`       										
                 ├── `roc_time.txt`
                 ├── `roc_time_perturb.txt`                    
              └── `invariance_test`
                 ├── `roc_synonym_substitute_perturb.txt`
                 ├── `roc_semantic_substitute_perturb.txt`
                 ├── `roc_contraction_perturb.txt`
                 ├── `roc_delete_punct_perturb.txt`
                 ├── `roc_typos_perturb.txt`
                 ├── `roc_negative_sample.txt`	# sampled negative samples from the discrimination test.	
                 ├── `roc_negative_sample_synonym_substitute_perturb.txt`
                 ├── `roc_negative_sample_semantic_substitute_perturb.txt`
                 ├── `roc_negative_sample_contraction_perturb.txt`
                 ├── `roc_negative_sample_delete_punct_perturb.txt`
                 ├── `roc_negative_sample_typos_perturb.txt`
       ├── `wp`
              └── ...

II. Tasks

OpenMEVA includes a suite of tasks to test comprehensive capabilities of metrics:

1. Correlation with human scores (based on MAGS)

2. Generalization across generation models and dataset (for learnable metrics, based on MAGS)

3. Judgment in general linguistic features (based on the discrimination test set of ACTS)

4. Robustness to rationality-preserving perturbations (based on the invariance test set of ACTS)

Note: The smaller absolute value of correlation is the better.

5. Fast Test

You can test these capabilities of new metrics by following command:

cd ./benchmark

# test correlation with human scores and generalization
python ./corr_gen.py

# test judgment
python ./judge.py

# test robustness
python ./robust.py

We take BLEU and Forward Perplexity as examples in the python files. You can test your own metrics by minor modification.

How to Cite

@misc{guan2021openmeva,
      title={OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics}, 
      author={Jian Guan and Zhexin Zhang and Zhuoer Feng and Zitao Liu and Wenbiao Ding and Xiaoxi Mao and Changjie Fan and Minlie Huang},
      year={2021},
      eprint={2105.08920},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

It's our honor to help you better explore language generation evaluation with our toolkit and benchmark.

Owner
Conversational AI groups from Tsinghua University
FID calculation with proper image resizing and quantization steps

clean-fid: Fixing Inconsistencies in FID Project | Paper The FID calculation involves many steps that can produce inconsistencies in the final metric.

Gaurav Parmar 606 Jan 06, 2023
Code release for "Conditional Adversarial Domain Adaptation" (NIPS 2018)

CDAN Code release for "Conditional Adversarial Domain Adaptation" (NIPS 2018) New version: https://github.com/thuml/Transfer-Learning-Library Dataset

THUML @ Tsinghua University 363 Dec 20, 2022
Demo project for real time anomaly detection using kafka and python

kafkaml-anomaly-detection Project for real time anomaly detection using kafka and python It's assumed that zookeeper and kafka are running in the loca

Rodrigo Arenas 36 Dec 12, 2022
A scikit-learn-compatible module for estimating prediction intervals.

|Anaconda|_ MAPIE - Model Agnostic Prediction Interval Estimator MAPIE allows you to easily estimate prediction intervals using your favourite sklearn

SimAI 584 Dec 27, 2022
Official code for the ICCV 2021 paper "DECA: Deep viewpoint-Equivariant human pose estimation using Capsule Autoencoders"

DECA Official code for the ICCV 2021 paper "DECA: Deep viewpoint-Equivariant human pose estimation using Capsule Autoencoders". All the code is writte

23 Dec 01, 2022
An open source app to help calm you down when needed.

By: Seanpm2001, Et; Al. Top README.md Read this article in a different language Sorted by: A-Z Sorting options unavailable ( af Afrikaans Afrikaans |

Sean P. Myrick V19.1.7.2 2 Oct 24, 2022
Omnidirectional Scene Text Detection with Sequential-free Box Discretization (IJCAI 2019). Including competition model, online demo, etc.

Box_Discretization_Network This repository is built on the pytorch [maskrcnn_benchmark]. The method is the foundation of our ReCTs-competition method

Yuliang Liu 266 Nov 24, 2022
Temporally Coherent GAN SIGGRAPH project.

TecoGAN This repository contains source code and materials for the TecoGAN project, i.e. code for a TEmporally COherent GAN for video super-resolution

Duc Linh Nguyen 2 Jan 18, 2022
Objax Apache-2Objax (🥉19 · ⭐ 580) - Objax is a machine learning framework that provides an Object.. Apache-2 jax

Objax Tutorials | Install | Documentation | Philosophy This is not an officially supported Google product. Objax is an open source machine learning fr

Google 729 Jan 02, 2023
Unofficial Pytorch Lightning implementation of Contrastive Syn-to-Real Generalization (ICLR, 2021)

Unofficial Pytorch Lightning implementation of Contrastive Syn-to-Real Generalization (ICLR, 2021)

Gyeongjae Choi 17 Sep 23, 2021
RRL: Resnet as representation for Reinforcement Learning

Resnet as representation for Reinforcement Learning (RRL) is a simple yet effective approach for training behaviors directly from visual inputs. We demonstrate that features learned by standard image

Meta Research 21 Dec 07, 2022
Code accompanying the NeurIPS 2021 paper "Generating High-Quality Explanations for Navigation in Partially-Revealed Environments"

Generating High-Quality Explanations for Navigation in Partially-Revealed Environments This work presents an approach to explainable navigation under

RAIL Group @ George Mason University 1 Oct 28, 2022
Code for the AI lab course 2021/2022 of the University of Verona

AI-Lab Code for the AI lab course 2021/2022 of the University of Verona Set-Up the environment for the curse Download Anaconda for your System. Instal

Davide Corsi 5 Oct 19, 2022
Fastquant - Backtest and optimize your trading strategies with only 3 lines of code!

fastquant 🤓 Bringing backtesting to the mainstream fastquant allows you to easily backtest investment strategies with as few as 3 lines of python cod

Lorenzo Ampil 1k Dec 29, 2022
Automatic Data-Regularized Actor-Critic (Auto-DrAC)

Auto-DrAC: Automatic Data-Regularized Actor-Critic This is a PyTorch implementation of the methods proposed in Automatic Data Augmentation for General

89 Dec 13, 2022
Learning Logic Rules for Document-Level Relation Extraction

LogiRE Learning Logic Rules for Document-Level Relation Extraction We propose to introduce logic rules to tackle the challenges of doc-level RE. Equip

41 Dec 26, 2022
FactSeg: Foreground Activation Driven Small Object Semantic Segmentation in Large-Scale Remote Sensing Imagery (TGRS)

FactSeg: Foreground Activation Driven Small Object Semantic Segmentation in Large-Scale Remote Sensing Imagery by Ailong Ma, Junjue Wang*, Yanfei Zhon

Kingdrone 43 Jan 05, 2023
Source code for the ACL-IJCNLP 2021 paper entitled "T-DNA: Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation" by Shizhe Diao et al.

T-DNA Source code for the ACL-IJCNLP 2021 paper entitled Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adapta

shizhediao 17 Dec 22, 2022
Streamlit App For Product Analysis - Streamlit App For Product Analysis

Streamlit_App_For_Product_Analysis Здравствуйте! Перед вами дашборд, позволяющий

Grigory Sirotkin 1 Jan 10, 2022