[ICCV 2021] Relaxed Transformer Decoders for Direct Action Proposal Generation

Last update: Nov 30, 2022

Overview

RTD-Net (ICCV 2021)

This repo holds the codes of paper: "Relaxed Transformer Decoders for Direct Action Proposal Generation", accepted in ICCV 2021.

News

[2021.8.17] We release codes, checkpoint and features on THUMOS14.

Overview

This paper presents a simple and end-to-end learnable framework (RTD-Net) for direct action proposal generation, by re-purposing a Transformer-alike architecture. Thanks to the parallel decoding of multiple proposals with explicit context modeling, our RTD-Net outperforms the previous state-of-the-art methods in temporal action proposal generation task on THUMOS14 and also yields a superior performance for action detection on this dataset. In addition, free of NMS post-processing, our detection pipeline is more efficient than previous methods.

Dependencies

Python 3.7 or higher
PyTorch 1.6 or higher
Torchvision
Numpy 1.19.2

Data Preparation

To reproduce the results in THUMOS14 without further changes:

Download the data from GoogleDrive.
Place I3D_features and TEM_scores into the folder data.

Checkpoint

Dataset	[email protected]	[email protected]	[email protected]	[email protected]	checkpoint
THUMOS14	41.52	49.33	56.41	62.91	link

Training

Use train.sh to train RTD-Net.


# First stage

CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 --master_port=11323 --use_env main.py --window_size 100 --batch_size 32 --stage 1 --num_queries 32 --point_prob_normalize

# Second stage for relaxation mechanism

CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 --master_port=11324 --use_env main.py --window_size 100 --batch_size 32 --lr 1e-5 --stage 2 --epochs 10 --lr_drop 5 --num_queries 32 --point_prob_normalize --load outputs/checkpoint_best_sum_ar.pth

# Third stage for completeness head

CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 --master_port=11325 --use_env main.py --window_size 100 --batch_size 32 --lr 1e-4 --stage 3 --epochs 20 --num_queries 32 --point_prob_normalize --load outputs/checkpoint_best_sum_ar.pth

Testing

Inference with test.sh.

CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 --master_port=11325 --use_env main.py --window_size 100 --batch_size 32 --lr 1e-4 --stage 3 --epochs 20 --num_queries 32 --point_prob_normalize --eval --resume outputs/checkpoint_best_sum_ar.pth

References

We especially thank the contributors of the BSN, G-TAD and DETR for providing helpful code.

Citations

If you think our work is helpful, please feel free to cite our paper.

@InProceedings{Tan_2021_RTD,
    author    = {Tan, Jing and Tang, Jiaqi and Wang, Limin and Wu, Gangshan},
    title     = {Relaxed Transformer Decoders for Direct Action Proposal Generation},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {13526-13535}
}

Contact

For any question, please file an issue or contact

Jing Tan: [email protected]
Jiaqi Tang: [email protected]

[ICCV 2021] Relaxed Transformer Decoders for Direct Action Proposal Generation

Related tags

Overview

RTD-Net (ICCV 2021)

News

Overview

Dependencies

Data Preparation

Checkpoint

Training

Testing

References

Citations

Contact

Owner

Multimedia Computing Group, Nanjing University

This repository is based on Ultralytics/yolov5, with adjustments to enable rotate prediction boxes.

Code for "Human Pose Regression with Residual Log-likelihood Estimation", ICCV 2021 Oral

A modern pure-Python library for reading PDF files

Towards Part-Based Understanding of RGB-D Scans

Asterisk is a framework to generate high-quality training datasets at scale

Benchmarks for the Optimal Power Flow Problem

A Physics-based Noise Formation Model for Extreme Low-light Raw Denoising (CVPR 2020 Oral & TPAMI 2021)

Official Repository for the paper "Improving Baselines in the Wild".

Regression Metrics Calculation Made easy for tensorflow2 and scikit-learn

Facilitates implementing deep neural-network backbones, data augmentations

You can draw the corresponding bounding box into the image and save it according to the result file (txt format) run by the tracker.

Code for Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Code and Experiments for ACL-IJCNLP 2021 Paper Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering.

HSC4D: Human-centered 4D Scene Capture in Large-scale Indoor-outdoor Space Using Wearable IMUs and LiDAR. CVPR 2022

An implementation for Neural Architecture Search with Random Labels (CVPR 2021 poster) on Pytorch.

DeepLabv3+：Encoder-Decoder with Atrous Separable Convolution语义分割模型在tensorflow2当中的实现

A toolkit for developing and comparing reinforcement learning algorithms.

Code for the paper titled "Prabhupadavani: A Code-mixed Speech Translation Data for 25 languages"

The GitHub repository for the paper: “Time Series is a Special Sequence: Forecasting with Sample Convolution and Interaction“.

Gradient-free global optimization algorithm for multidimensional functions based on the low rank tensor train format