Code for the RA-L (ICRA) 2021 paper "SeqNet: Learning Descriptors for Sequence-Based Hierarchical Place Recognition"

Last update: Dec 12, 2022

Overview

SeqNet: Learning Descriptors for Sequence-Based Hierarchical Place Recognition

[ArXiv+Supplementary] [IEEE Xplore RA-L 2021] [ICRA 2021 YouTube Video]

and

SeqNetVLAD vs PointNetVLAD: Image Sequence vs 3D Point Clouds for Day-Night Place Recognition

[ArXiv] [CVPR 2021 Workshop 3DVR]

Sequence-Based Hierarchical Visual Place Recognition.

News:

Jun 23: CVPR 2021 Workshop 3DVR paper, "SeqNetVLAD vs PointNetVLAD", now available on arXiv. Oxford dataset to be released soon.

Jun 02: SeqNet code release with the Nordland dataset.

Setup (One time)

Conda

conda create -n seqnet python=3.8 mamba -c conda-forge -y
conda activate seqnet
mamba install numpy pytorch=1.8.0 torchvision tqdm scikit-learn faiss tensorboardx h5py -c conda-forge -y

Download

Run bash download.sh to download single image NetVLAD descriptors (3.4 GB) for the Nordland-clean dataset ^[a] and corresponding model files (1.5 GB) ^[b].

Run

Train

To train sequential descriptors through SeqNet:

python main.py --mode train --pooling seqnet --dataset nordland-sw --seqL 10 --w 5 --outDims 4096 --expName "w5"

To (re-)train single descriptors through SeqNet:

python main.py --mode train --pooling seqnet --dataset nordland-sw --seqL 1 --w 1 --outDims 4096 --expName "w1"

Test

python main.py --mode test --pooling seqnet --dataset nordland-sf --seqL 5 --split test --resume ./data/runs/Jun03_15-22-44_l10_w5/

The above will reproduce results for SeqNet (S5) as per Supp. Table III on Page 10.

To obtain other results from the same table, expand this.

# Raw Single (NetVLAD) Descriptor
python main.py --mode test --pooling single --dataset nordland-sf --seqL 1 --split test

# SeqNet (S1)
python main.py --mode test --pooling seqnet --dataset nordland-sf --seqL 1 --split test --resume ./data/runs/Jun03_15-07-46_l1_w1/

# Raw + Smoothing
python main.py --mode test --pooling smooth --dataset nordland-sf --seqL 5 --split test

# Raw + Delta
python main.py --mode test --pooling delta --dataset nordland-sf --seqL 5 --split test

# Raw + SeqMatch
python main.py --mode test --pooling single+seqmatch --dataset nordland-sf --seqL 5 --split test

# SeqNet (S1) + SeqMatch
python main.py --mode test --pooling s1+seqmatch --dataset nordland-sf --seqL 5 --split test --resume ./data/runs/Jun03_15-07-46_l1_w1/

# HVPR (S5 to S1)
# Run S5 first and save its predictions by specifying `resultsPath`
python main.py --mode test --pooling seqnet --dataset nordland-sf --seqL 5 --split test --resume ./data/runs/Jun03_15-22-44_l10_w5/ --resultsPath ./data/results/
# Now run S1 + SeqMatch using results from above (the timestamp of `predictionsFile` would be different in your case)
python main.py --mode test --pooling s1+seqmatch --dataset nordland-sf --seqL 5 --split test --resume ./data/runs/Jun03_15-07-46_l1_w1/ --predictionsFile ./data/results/Jun03_16-07-36_l5_0.npz

Acknowledgement

The code in this repository is based on Nanne/pytorch-NetVlad. Thanks to Tobias Fischer for his contributions to this code during the development of our project QVPR/Patch-NetVLAD.

Citation

@article{garg2021seqnet,
  title={SeqNet: Learning Descriptors for Sequence-based Hierarchical Place Recognition},
  author={Garg, Sourav and Milford, Michael},
  journal={IEEE Robotics and Automation Letters},
  volume={6},
  number={3},
  pages={4305-4312},
  year={2021},
  publisher={IEEE},
  doi={10.1109/LRA.2021.3067633}
}

@misc{garg2021seqnetvlad,
  title={SeqNetVLAD vs PointNetVLAD: Image Sequence vs 3D Point Clouds for Day-Night Place Recognition},
  author={Garg, Sourav and Milford, Michael},
  howpublished={CVPR 2021 Workshop on 3D Vision and Robotics (3DVR)},
  month={Jun},
  year={2021},
}

Other Related Projects

Patch-NetVLAD (2021); Delta Descriptors (2020); CoarseHash (2020); seq2single (2019); LoST (2018)

[a] This is the clean version of the dataset that excludes images from the tunnels and red lights, exact image names can be obtained from here.

[b] These will automatically save to ./data/, you can modify this path in download.sh and get_datasets.py to specify your workdir.

Code for the RA-L (ICRA) 2021 paper "SeqNet: Learning Descriptors for Sequence-Based Hierarchical Place Recognition"

Related tags

Overview

SeqNet: Learning Descriptors for Sequence-Based Hierarchical Place Recognition

SeqNetVLAD vs PointNetVLAD: Image Sequence vs 3D Point Clouds for Day-Night Place Recognition

News:

Setup (One time)

Conda

Download

Run

Train

Test

Acknowledgement

Citation

Other Related Projects

Owner

Sourav Garg

Visual Question Answering in Pytorch

Combine Tacotron2 and Hifi GAN to generate speech from text

A Pytorch implementation of SMU: SMOOTH ACTIVATION FUNCTION FOR DEEP NETWORKS USING SMOOTHING MAXIMUM TECHNIQUE

Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

EMNLP 2021 paper The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers.

Leaderboard and Visualization for RLCard

Combinatorial model of ligand-receptor binding

Code for "ATISS: Autoregressive Transformers for Indoor Scene Synthesis", NeurIPS 2021

DeceFL: A Principled Decentralized Federated Learning Framework

Sharpness-Aware Minimization for Efficiently Improving Generalization

[CVPR 2019 Oral] Multi-Channel Attention Selection GAN with Cascaded Semantic Guidance for Cross-View Image Translation

Optimizers-visualized - Visualization of different optimizers on local minimas and saddle points.

A unified framework to jointly model images, text, and human attention traces.

Code for "Adversarial Attack Generation Empowered by Min-Max Optimization", NeurIPS 2021

Learning the Beauty in Songs: Neural Singing Voice Beautifier; ACL 2022 (Main conference); Official code

4th place solution to datafactory challenge by Intermarché.

For medical image segmentation

PyTorch implementation of "Conformer: Convolution-augmented Transformer for Speech Recognition" (INTERSPEECH 2020)

Official PyTorch implementation of UACANet: Uncertainty Aware Context Attention for Polyp Segmentation

Vpw analyzer - A visual J1850 VPW analyzer written in Python