Reformulation-Aware-Metrics

Introduction

This codebase contains source-code of the Python-based implementation of our CIKM 2021 paper.

• Chen, Jia, et al. "Incorporating Query Reformulating Behavior into Web Search Evaluation." To Appear in Proceedings of the 30th ACM International Conference on Information and Knowledge Management. 2021.

Requirements

• python 2.7
• sklearn
• scipy

Data Preparation

Preprocess two datasets TianGong-SS-FSD and TianGong-Qref into the the following format:

[Reformulation Type][Click List][Usefulness List][Satisfaction Label]

• Reformulation Type: A (Add), D (Delete), K (Keep), T (Transform or Change), O (Others), F (First Query).
• Click List: 1 -- Clicked, 0 -- Not Clicked.
• Usefulness List: Usefulness or Relevance, 4-scale in TianGong-QRef, 5-scale in TianGong-SS-FSD.
• Satisfaction Label: 5-scale for both datasets.

Then, bootsrap them into N samples and put the bootstapped data (directories) into ./data/bootstrap_fsd and ./data/bootstrap_qref.

Results

The results for each metrics are shown in the following table:

To reproduce the results of traditional metrics such as RBP, DCG and BPM, we recommend you to use this repo: cwl_eval. 🤗

Quick Start

To train RAMs, run the script as follows:

python run.py --click_model DBN \
	--data qref \
	--id 0 \
	--metric_type expected_utility \
	--max_usefulness 3 \
	--k_num 6 \
	--max_dnum 10 \
	--iter_num 10000 \
	--alpha 0.01 \
	--alpha_decay 0.99 \
	--lamda 0.85 \
	--patience 5 \
	--use_knowledge True

• click_model: options: ['DBN', 'SDBN', 'UBM', 'PBM']
• data: options: ['fsd', 'qref']
• metric_type: options: ['expected_utility', 'effort']
• id: the bootstrapped sample id.
• k_num: the number of user intent shift type will be considered, should be less than or equal to six.
• max_dnum: the maximum number of top documents to be considered for a specific query.
• use_knowledge: whether to use the transition probability from syntactic reformulation types to intent-level ones derived from the TianGong-Qref dataset.

Citation

If you find the resources in this repo useful, please do not save your star and cite our work:

@inproceedings{chen2021incorporating,
  title={Incorporating Query Reformulating Behavior into Web Search Evaluation},
  author={Chen, Jia and Liu, Yiqun and Mao, Jiaxin and Zhang, Fan and Sakai, Tetsuya and Ma, Weizhi and Zhang, Min and Ma, Shaoping},
  booktitle={Proceedings of the 30th ACM International Conference on Information and Knowledge Management},
  year={2021},
  organization={ACM}
}

Contact

If you have any questions, please feel free to contact me via [email protected] or open an issue.