RNG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering

Related tags

Deep Learningrng-kbqa
Overview

RNG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering

Authors: Xi Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou and Caiming Xiong

Abstract

main figure

Existing KBQA approaches, despite achieving strong performance on i.i.d. test data, often struggle in generalizing to questions involving unseen KB schema items. Prior rankingbased approaches have shown some success in generalization, but suffer from the coverage issue. We present RnG-KBQA, a Rank-andGenerate approach for KBQA, which remedies the coverage issue with a generation model while preserving a strong generalization capability. Our approach first uses a contrastive ranker to rank a set of candidate logical forms obtained by searching over the knowledge graph. It then introduces a tailored generation model conditioned on the question and the top-ranked candidates to compose the final logical form. We achieve new state-ofthe-art results on GRAILQA and WEBQSP datasets. In particular, our method surpasses the prior state-of-the-art by a large margin on the GRAILQA leaderboard. In addition, RnGKBQA outperforms all prior approaches on the popular WEBQSP benchmark, even including the ones that use the oracle entity linking. The experimental results demonstrate the effectiveness of the interplay between ranking and generation, which leads to the superior performance of our proposed approach across all settings with especially strong improvements in zero-shot generalization.

Paper link: https://arxiv.org/pdf/2109.08678.pdf

Requirements

The code is tested under the following environment setup

  • python==3.8.10
  • pytorch==1.7.0
  • transformers==3.3.1
  • spacy==3.1.1
  • other requirments please see requirements.txt

System requirements:

It's recommended to use a machine with over 300G memory to train the models, and use a machine with 128G memory for inference. However, 256G memory will still be sufficient for runing inference and training all of the models (some tricks for saving memorry is needed in training ranker model for GrailQA).

General Setup

Setup Experiment Directory

Before Running the scripts, please use the setup.sh to setup the experiment folder. Basically it creates some symbolic links in each exp directory.

Setup Freebase

All of the datasets use Freebase as the knowledge source. Please follow Freebase Setup to set up a Virtuoso triplestore service. If you modify the default url, you may need to change the url in /framework/executor/sparql_executor.py accordingly, after starting your virtuoso service,

Reproducing the Results on GrailQA

Please use /GrailQA as the working directory when running experiments on GrailQA.


Prepare dataset and pretrained checkpoints

Dataset

Please download the dataset and put the them under outputs so as to organize dataset as outputs/grailqa_v1.0_train/dev/test.json. (Please rename test-public split to test split).

NER Checkpoints

We use the NER system (under directory entity_linking and entity_linker) from Original GrailQA Code Repo. Please use the following instructions (copied from oringinal repo) to pull related data

Other Checkpoints

Please download the following checkpoints for entity disambiguation, candidate ranking, and augmented generation checkpoints, unzip and put them under checkpoints/ directory

KB Cache

We attach the cache of query results from KB, which can help save some time. Please download the cache file for grailqa, unzip and put them under cache/, so that we have cache/grail-LinkedRelation.bin and cache/grail-TwoHopPath.bin in the place.


Running inference

Demo for Checking the Pipeline

It's recommended to use the one-click demo scripts first to test if everything mentioned above is setup correctly. If it successfully run through, you'll get a final F1 of around 0.86. Please make sure you successfully reproduce the results on this small demo set first, as inference on dev and test can take a long time.

sh scripts/walk_through_demo.sh

Step by Step Instructions

We also provide step-by-step inference instructions as below:

(i) Detecting Entities

Once having the entity linker ready, run

python detect_entity_mention.py --split # eg. --split test

This will write entity mentions to outputs/grail_ _entities.json , we extract up to 10 entities for each mention, which will be further disambiguate in the next step.

!! Running entity detection for the first time will require building surface form index, which can take a long time (but it's only needed for the first time).

(ii) Disambiguating Entities (Entity Linking)

We have provided pretrained ranker model

sh scripts/run_disamb.sh predict

E.g., sh scripts/run_disamb.sh predict checkpoints/grail_bert_entity_disamb test

This will write the prediction results (in the form of selected entity index for each mention) to misc/grail_ _entity_linking.json .

(iii) Enumerating Logical Form Candidates

python enumerate_candidates.py --split --pred_file

E.g., python enumerate_candidates.py --split test --pred_file misc/grail_test_entity_linking.json.

This will write enumerated candidates to outputs/grail_ _candidates-ranking.jsonline .

(iv) Running Ranker

sh scripts/run_ranker.sh predict

E.g., sh scripts/run_ranker.sh predict checkpoints/grail_bert_ranking test

This will write prediction candidate logits (the logits of each candidate for each example) to misc/grail_ _candidates_logits.bin , and prediction result (in original GrailQA prediction format) to misc/grail_ _ranker_results.txt

You may evaluate the ranker results by python grail_evaluate.py

E.g., python grail_evaluate.py outputs/grailqa_v1.0_dev.json misc/grail_dev_ranker_results.txt

(v) Running Generator

First, make prepare generation model inputs

python make_generation_dataset.py --split --logit_file

E.g., python make_generation_dataset.py --split test --logit_file misc/grail_test_candidate_logits.bin.

This will read the canddiates and the use logits to select top-k candidates and write generation model inputs to outputs/grail_ _gen.json .

Second, run generation model to get the top-k prediction

sh scripts/run_gen.sh predict

E.g., sh scripts/run_gen.sh predict checkpoints/grail_t5_generation test.

This will generate top-k decoded logical forms stored at misc/grail_ _topk_generations.json .

(vi) Final Inference Steps

Having the decoded top-k predictions, we'll go down the top-k list, execute the logical form one by one until we find one logical form return valid answers.

python eval_topk_prediction.py --split --pred_file

E.g., python eval_topk_prediction.py --split test --pred_file misc/grail_test_topk_generations.json

prediction result (in original GrailQA prediction format) to misc/grail_ _final_results.txt .

You can then use official GrailQA evaluate script to run evaluation

python grail_evaluate.py

E.g., python grail_evaluate.py outputs/grailqa_v1.0_dev.json misc/grail_dev_final_results.txt


Training Models

We already attached pretrained-models ready for running inference. If you'd like to train your own models please checkout the README at /GrailQA folder.

Reproducing the Results on WebQSP

Please use /WebQSP as the working directory when running experiments on WebQSP.


Prepare dataset and pretrained checkpoints

Dataset

Please download the WebQSP dataset and put the them under outputs so as to organize dataset as outputs/WebQSP.train[test].json.

Evaluation Script

Please make a copy of the official evaluation script (eval/eval.py in the WebQSP zip file) and put the script under this directory (WebQSP) with the name legacy_eval.py.

Model Checkpoints

Please download the following checkpoints for candidate ranking, and augmented generation checkpoints, unzip and put them under checkpoints/ directory

KB Cache

Please download the cache file for webqsp, unzip and put them under cache/ so that we have cache/webqsp-LinkedRelation.bin and cache/webqsp-TwoHopPath.bin in the place.


Running inference

(i) Parsing Sparql-Query to S-Expression

As stated in the paper, we generate s-expressions, which is not provided by the original dataset, so we provide scripts to parse sparql-query to s-expressions.

Run python parse_sparql.py, which will augment original dataset files with s-expressions and save them in outputs as outputs/WebQSP.train.expr.json and outputs/WebQSP.dev.expr.json. Since there is no validation set, we further randomly select 200 examples from the training set for validation, yielding ptrain split and pdev split.

(ii) Entity Detection and Linking using ELQ

This step can be skipped, as we've already include outputs of this step (misc/webqsp_train_elq-5_mid.json, outputs/webqsp_test_elq-5_mid.json).

The scripts and config of ELQ model can be found in elq_linking/run_elq_linker.py. If you'd like to use the script to run entity linking, please copy the run_elq_linker.py python script to ELQ model and run the script there.

(iii) Enumerating Logical Form Candidates

python enumerate_candidates.py --split test

This will write enumerated candidates to outputs/webqsp_test_candidates-ranking.jsonline.

(iv) Runing Ranker

sh scripts/run_ranker.sh predict checkpoints/webqsp_bert_ranking test

This will write prediction candidate logits (the logits of each candidate for each example) to misc/webqsp_test_candidates_logits.bin, and prediction result (in original GrailQA prediction format) to misc/webqsp_test_ranker_results.txt

(v) Running Generator

First, make prepare generation model inputs

python make_generation_dataset.py --split test --logit_file misc/webqsp_test_candidate_logits.bin.

This will read the candidates and the use logits to select top-k candidates and write generation model inputs to outputs/webqsp_test_gen.json.

Second, run generation model to get the top-k prediction

sh scripts/run_gen.sh predict checkpoints/webqsp_t5_generation test

This will generate top-k decoded logical forms stored at misc/webqsp_test_topk_generations.json.

(vi) Final Inference Steps

Having the decoded top-k predictions, we'll go down the top-k list, execute the logical form one by one until we find one logical form return valid answers.

python eval_topk_prediction.py --split test --pred_file misc/webqsp_test_topk_generations.json

Prediction result will be stored (in GrailQA prediction format) to misc/webqsp_test_final_results.txt.

You can then use official WebQSP (only modified in I/O) evaluate script to run evaluation

python webqsp_evaluate.py outputs/WebQSP.test.json misc/webqsp_test_final_results.txt.


Training Models

We already attached pretrained-models ready for running inference. If you'd like to train your own models please checkout the README at /WebQSP folder.

Citation

@misc{ye2021rngkbqa,
    title={RnG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering}, 
    author={Xi Ye and Semih Yavuz and Kazuma Hashimoto and Yingbo Zhou and Caiming Xiong},
    year={2021},
    eprint={2109.08678},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

Questions?

For any questions, feel free to open issues, or shoot emails to

License

The code is released under BSD 3-Clause - see LICENSE for details.

Owner
Salesforce
A variety of vendor agnostic projects which power Salesforce
Salesforce
Code to produce syntactic representations that can be used to study syntax processing in the human brain

Can fMRI reveal the representation of syntactic structure in the brain? The code base for our paper on understanding syntactic representations in the

Aniketh Janardhan Reddy 4 Dec 18, 2022
Code for our CVPR 2021 Paper "Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes".

Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes (CVPR 2021) Project page | Paper | Colab | Colab for Drawing App Rethinking Style

CompVis Heidelberg 153 Jan 04, 2023
SMPL-X: A new joint 3D model of the human body, face and hands together

SMPL-X: A new joint 3D model of the human body, face and hands together [Paper Page] [Paper] [Supp. Mat.] Table of Contents License Description News I

Vassilis Choutas 1k Jan 09, 2023
Code for our ICASSP 2021 paper: SA-Net: Shuffle Attention for Deep Convolutional Neural Networks

SA-Net: Shuffle Attention for Deep Convolutional Neural Networks (paper) By Qing-Long Zhang and Yu-Bin Yang [State Key Laboratory for Novel Software T

Qing-Long Zhang 199 Jan 08, 2023
Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

The Hypersim Dataset For many fundamental scene understanding tasks, it is difficult or impossible to obtain per-pixel ground truth labels from real i

Apple 1.3k Jan 04, 2023
Implementation of EMNLP 2017 Paper "Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog" using PyTorch and ParlAI

Language Emergence in Multi Agent Dialog Code for the Paper Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog Satwik Kottur, José M.

Karan Desai 105 Nov 25, 2022
GANSketchingJittor - Implementation of Sketch Your Own GAN in Jittor

GANSketching in Jittor Implementation of (Sketch Your Own GAN) in Jittor(计图). Or

Bernard Tan 10 Jul 02, 2022
Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP

Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP Abstract: We introduce a method that allows to automatically se

Daniil Pakhomov 134 Dec 19, 2022
Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory"

Memory Efficient Attention Pytorch Implementation of a memory efficient multi-head attention as proposed in the paper, Self-attention Does Not Need O(

Phil Wang 180 Jan 05, 2023
This repository contains all code and data for the Inside Out Visual Place Recognition task

Inside Out Visual Place Recognition This repository contains code and instructions to reproduce the results for the Inside Out Visual Place Recognitio

15 May 21, 2022
External Attention Network

Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks paper : https://arxiv.org/abs/2105.02358 EAMLP will come soon Jitto

MenghaoGuo 357 Dec 11, 2022
Code for NeurIPS 2021 paper 'Spatio-Temporal Variational Gaussian Processes'

Spatio-Temporal Variational GPs This repository is the official implementation of the methods in the publication: O. Hamelijnck, W.J. Wilkinson, N.A.

AaltoML 26 Sep 16, 2022
ConformalLayers: A non-linear sequential neural network with associative layers

ConformalLayers: A non-linear sequential neural network with associative layers ConformalLayers is a conformal embedding of sequential layers of Convo

Prograf-UFF 5 Sep 28, 2022
AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty

AugMix Introduction We propose AugMix, a data processing technique that mixes augmented images and enforces consistent embeddings of the augmented ima

Google Research 876 Dec 17, 2022
Meandering In Networks of Entities to Reach Verisimilar Answers

MINERVA Meandering In Networks of Entities to Reach Verisimilar Answers Code and models for the paper Go for a Walk and Arrive at the Answer - Reasoni

Shehzaad Dhuliawala 271 Dec 13, 2022
Using the provided dataset which includes various book features, in order to predict the price of books, using various proposed methods and models.

Using the provided dataset which includes various book features, in order to predict the price of books, using various proposed methods and models.

Nikolas Petrou 1 Jan 13, 2022
Zalo AI challenge 2021 task hum to song

Zalo AI challenge 2021 task Hum to Song pipeline: Chuẩn bị dữ liệu cho quá trình train: Sửa các file đường dẫn trong config/preprocess.yaml raw_path:

Vo Van Phuc 105 Dec 16, 2022
Framework for Spectral Clustering on the Sparse Coefficients of Learned Dictionaries

Dictionary Learning for Clustering on Hyperspectral Images Overview Framework for Spectral Clustering on the Sparse Coefficients of Learned Dictionari

Joshua Bruton 6 Oct 25, 2022
Implementation of Geometric Vector Perceptron, a simple circuit for 3d rotation equivariance for learning over large biomolecules, in Pytorch. Idea proposed and accepted at ICLR 2021

Geometric Vector Perceptron Implementation of Geometric Vector Perceptron, a simple circuit with 3d rotation equivariance for learning over large biom

Phil Wang 59 Nov 24, 2022
The official code of Anisotropic Stroke Control for Multiple Artists Style Transfer

ASMA-GAN Anisotropic Stroke Control for Multiple Artists Style Transfer Proceedings of the 28th ACM International Conference on Multimedia The officia

Six_God 146 Nov 21, 2022