HCQ: Hybrid Contrastive Quantization for Efficient Cross-View Video Retrieval

[toc]

1. Introduction

This repository provides the code for our paper at TheWebConf 2022:

Hybrid Contrastive Quantization for Efficient Cross-View Video Retrieval. Jinpeng Wang, Bin Chen, Dongliang Liao, Ziyun Zeng, Gongfu Li, Shu-Tao Xia, Jin Xu. [arXiv].

Our proposed Hybrid Contrastive Quantization (HCQ) is the first quantization learning method for cross-view (e.g., text-to-video) retrieval, which learns both coarse-grained and fine-grained quantizations with transformers. Experiments on MSRVTT, LSMDC and ActivityNet Captions datasets demonstrate that it can achieve competitive performance with state-of-the-art non-compressed retrieval methods while showing high efficiency in storage and computation.

In the following, we will guide you how to use this repository step by step. 🤗

2. Preparation

git clone https://github.com/gimpong/WWW22-HCQ.git

2.1 Requirements

• python 3.7.4
• gensim 4.1.2
• h5py 3.6.0
• numpy 1.17.3
• pandas 1.2.3
• pytorch-warmup 0.0.4
• scikit-learn 0.23.0
• scipy 1.6.1
• tensorboardX 2.4.1
• torch 1.6.0+cu101
• transformers 3.1.0

cd WWW22-HCQ
# Install the requirements
pip install -r requirements.txt

We conduct each training experiment on a single NVIDIA® Tesla® V100 GPU (32 GB).

2.2 Download the features

Before running the code, we need to download the datasets and arrange them in the "data" directory properly. We use the video features provided by the authors of MMT. These features can be downloaded from this page by running the following commands:

# Create and move to WWW22-HCQ/data directory
cd data
# Download the video features
wget http://pascal.inrialpes.fr/data2/vgabeur/video-features/MSRVTT.tar.gz
wget http://pascal.inrialpes.fr/data2/vgabeur/video-features/activity-net.tar.gz
wget http://pascal.inrialpes.fr/data2/vgabeur/video-features/LSMDC.tar.gz
# Extract the video features
tar -xvf MSRVTT.tar.gz
tar -xvf activity-net.tar.gz
tar -xvf LSMDC.tar.gz

3. Training and Evaluation

3.1 Training from scratch

Let us take "training HCQ on MSRVTT dataset ('1k-A' split)" as an example:

# working directory: WWW22-HCQ/
python -m train --config configs/HCQ_MSRVTT_1kA.json

Expected results:

MSRVTT_jsfusion_test:
 t2v_metrics/R1/final_eval: 25.9
 t2v_metrics/R5/final_eval: 54.8
 t2v_metrics/R10/final_eval: 69.0
 t2v_metrics/R50/final_eval: 88.8
 t2v_metrics/MedR/final_eval: 5.0
 t2v_metrics/MeanR/final_eval: 28.062
 t2v_metrics/geometric_mean_R1-R5-R10/final_eval: 46.09386629981193
 v2t_metrics/R1/final_eval: 26.3
 v2t_metrics/R5/final_eval: 57.0
 v2t_metrics/R10/final_eval: 70.1
 v2t_metrics/R50/final_eval: 90.0
 v2t_metrics/MedR/final_eval: 4.0
 v2t_metrics/MeanR/final_eval: 25.1535
 v2t_metrics/geometric_mean_R1-R5-R10/final_eval: 47.18995255588879

After training, a folder with the same name as the configuration json file (e.g., "HCQ_MSRVTT_1kA") will be generated under WWW22-HCQ/exps/, which contains the model checkpoints, logs, tensorboard files, and so on.

For reproducing other experiments, please see the following tables. You can just replace the config json path with another in the training command.

3.1.1 Main results of HCQ (reported in Table 1-3 in our paper)

3.1.2 Result of Hybrid Contrastive Transformer (HCT), Dual Transformer (DT) + DCMH, and DT + JPQ (reported in Table 4 in our paper)

3.1.3 Results of HCQ under different hyper-parameters (reported in Figure 6 in our paper)

3.1.4 Results of HCQ with different kinds of text encoders ("1k-A" split) (reported in Table 5 in our paper)

If you are doing experiments on a platform with enough RAM and want to accelerate the training, you can load the whole dataset in RAM by the following modification:

# WWW22-HCQ/base/base_dataset.py:L170
               load_in_ram=True, # change from 'False' to 'True'

3.2 Evaluation from checkpoint

We can evaluate the model from the checkpoint without re-training. The evaluation command:

python -m train --config configs/HCQ_MSRVTT_1kA.json --only_eval --load_checkpoint HCQ_MSRVTT_1kA.pth

We provide the checkpoint of HCQ_MSRVTT_1kA.json as an example, you can download this file (~1.6G) from the Google Drive and put it in the working directory (WWW22-HCQ/).

3.3 Evaluation for post-compression methods

Take the evaluation on MSRVTT dataset ("1k-A" split) as an example. First, we need to train an HCT.

# working directory: WWW22-HCQ/
python -m train --config configs/HCT_MSRVTT_1kA.json

Then, run the get_embed.py and pass the path of the HCT checkpoint to the script:

python -m get_embed configs/HCT_MSRVTT_1kA.json --only_eval --load_checkpoint HCT_MSRVTT_1kA/trained_model.pth

After that, we will get the embedding file embeddings.h5 under WWW22-HCQ/exps/HCT_MSRVTT_1kA/. Run the compress_embed.py and get the results:

# compress embeddings with LSH
python -m compress_embed --path ./exps/HCT_MSRVTT_1kA/embeddings.h5 --type LSH
# compress embeddings with PQ
python -m compress_embed --path ./exps/HCT_MSRVTT_1kA/embeddings.h5 --type PQ
# compress embeddings with OPQ
python -m compress_embed --path ./exps/HCT_MSRVTT_1kA/embeddings.h5 --type OPQ

3. References

If you find this code useful or use the toolkit in your work, please consider citing:

@inproceedings{wang22hcq,
  author={Wang, Jinpeng and Chen, Bin and Liao, Dongliang and Zeng, Ziyun and Li, Gongfu and Shu-Tao, Xia and Xu, Jin},
  title={Hybrid Contrastive Quantization for Efficient Cross-View Video Retrieval},
  booktitle={Proceedings of the Web Conference 2022},
  doi={10.1145/3485447.3512022}
}

4. Acknowledgements

Our code is based on the implementation of nanopq, Multi-Modal Transformer, Collaborative Experts, Transformers and Mixture of Embedding Experts.

5. Contact

If you have any question, you can raise an issue or email Jinpeng Wang ([email protected]). We will reply you soon.