This is the code for the paper "Jinkai Zheng, Xinchen Liu, Wu Liu, Lingxiao He, Chenggang Yan, Tao Mei: Gait Recognition in the Wild with Dense 3D Representations and A Benchmark. (CVPR 2022)"

Last update: Jan 04, 2023

Overview

Gait3D-Benchmark

This is the code for the paper "Jinkai Zheng, Xinchen Liu, Wu Liu, Lingxiao He, Chenggang Yan, Tao Mei: Gait Recognition in the Wild with Dense 3D Representations and A Benchmark. (CVPR 2022)". The official project page is here.

What's New

[Mar 2022] Another gait in the wild dataset GREW is supported.
[Mar 2022] Our Gait3D dataset and SMPLGait method are released.

Model Zoo

Gait3D

Input Size: 128x88(64x44)

Method	[email protected]	[email protected]	mAP	mINP	download
GaitSet(AAAI2019))	42.60(36.70)	63.10(58.30)	33.69(30.01)	19.69(17.30)	model-128(model-64)
GaitPart(CVPR2020)	29.90(28.20)	50.60(47.60)	23.34(21.58)	13.15(12.36)	model-128(model-64)
GLN(ECCV2020)	42.20(31.40)	64.50(52.90)	33.14(24.74)	19.56(13.58)	model-128(model-64)
GaitGL(ICCV2021)	23.50(29.70)	38.50(48.50)	16.40(22.29)	9.20(13.26)	model-128(model-64)
OpenGait Baseline*	47.70(42.90)	67.20(63.90)	37.62(35.19)	22.24(20.83)	model-128(model-64)
SMPLGait(CVPR2022)	53.20(46.30)	71.00(64.50)	42.43(37.16)	25.97(22.23)	model-128(model-64)

*It should be noticed that OpenGait Baseline is equal to SMPLGait w/o 3D in our paper.

Cross Domain

Datasets in the Wild (GaitSet, 64x44)

Source	Target	[email protected]	[email protected]	mAP
GREW (official split)	Gait3D	15.80	30.20	11.83
GREW (our split)	Gait3D	16.50	31.10	11.71
Gait3D	GREW (official split)	18.81	32.25	~
Gait3D	GREW (our split)	43.86	60.89	28.06

Requirements

pytorch >= 1.6
torchvision
pyyaml
tensorboard
opencv-python
tqdm
py7zr
tabulate
termcolor

Installation

You can replace the second command from the bottom to install pytorch based on your CUDA version.

git clone https://github.com/Gait3D/Gait3D-Benchmark.git
cd Gait3D-Benchmark
conda create --name py37torch160 python=3.7
conda activate py37torch160
conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.2 -c pytorch
pip install tqdm pyyaml tensorboard opencv-python tqdm py7zr tabulate termcolor

Data Preparation

Please download the Gait3D dataset by signing an agreement. We ask for your information only to make sure the dataset is used for non-commercial purposes. We will not give it to any third party or publish it publicly anywhere.

Data Pretreatment

Run the following command to preprocess the Gait3D dataset.

python misc/pretreatment.py --input_path 'Gait3D/2D_Silhouettes' --output_path 'Gait3D-sils-64-44-pkl' --img_h 64 --img_w 44
python misc/pretreatment.py --input_path 'Gait3D/2D_Silhouettes' --output_path 'Gait3D-sils-128-88-pkl' --img_h 128 --img_w 88
python misc/pretreatment_smpl.py --input_path 'Gait3D/3D_SMPLs' --output_path 'Gait3D-smpls-pkl'

Data Structrue

After the pretreatment, the data structure under the directory should like this

├── Gait3D-sils-64-44-pkl
│  ├── 0000
│     ├── camid0_videoid2
│        ├── seq0
│           └──seq0.pkl
├── Gait3D-sils-128-88-pkl
│  ├── 0000
│     ├── camid0_videoid2
│        ├── seq0
│           └──seq0.pkl
├── Gait3D-smpls-pkl
│  ├── 0000
│     ├── camid0_videoid2
│        ├── seq0
│           └──seq0.pkl

Train

Run the following command:

sh train.sh

Test

Run the following command:

sh test.sh

Citation

Please cite this paper in your publications if it helps your research:

@inproceedings{zheng2022gait3d,
  title={Gait Recognition in the Wild with Dense 3D Representations and A Benchmark},
  author={Jinkai Zheng, Xinchen Liu, Wu Liu, Lingxiao He, Chenggang Yan, Tao Mei},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2022}
}

Acknowledgement

Here are some great resources we benefit:

The codebase is based on OpenGait.
The 3D SMPL data is obtained by ROMP.
The 2D Silhouette data is obtained by HRNet-segmentation.
The 2D pose data is obtained by HRNet.
The ReID featrue used to make Gait3D is obtained by FastReID.

Comments

lib/modeling/models/smplgait.py throwing error when training a new dataset

Hi Jinkai,

When I try to use the SMPLGait to apply on other dataset, during the training process, the smplgait.py throws the error that: smpls = ipts[1][0] # [n, s, d] IndexError: list index out of range It is also interesting that I used 4 GPUs in the training. 3 of them could detect the the ipts[1][0] tensor with size 1. However, the fourth one failed to do so. Could I know how I can solve this?

opened by zhiyuann 7
I have a few questions about Gait3D-Benchmark Datasets

Hi. Im jjun. I read your paper impressively.

We don't currently live in China, so it is difficult to use dataset on baidu disk.

If you don't mind, is there a way to download the dataset to another disk (e.g Google drive)?

opened by jjunnii 6
Question about 3D SMPL skeleton topology diagram

Your work promotes the application of gait recognition in real scenes, can you provide the topology diagram of the SMPL 3D skeleton in Gait3D? Because the specific meaning of the 24 joint points is not stated in your data description document.

opened by HL-HYX 4
ROMP SMPL transfer

When I try to use the ROMP to generate out the 3D mesh, I detect there is a version conflict with the ROMP used by SMPLGait. Could I know which version of the ROMP the SMPLGait used? In this way I could use the SMPLGait to run on other ReID dataset.

opened by zhiyuann 3
question about iteration and epoch

Hi! The total iteration in your code is set to 180000, and you report the total epoch as 1200 in your paper. What's the relationship between iteration and epoch?

opened by yan811 2
About data generation

Hi! I 'd like to know some details about data generation in NPZ files.

In npz file: 1 What's the order of "pose"? SMPL pose parameter should be [24,3] dim, how did you convert it to [72,]? The order is [keypoint1_angel1, keypoint1_angle2, keypoint1_angle3, keypoint2_angel1, keypoint2_angle2, keypoint2_angle3...] or [keypoint1_angle1, keypoint2_angle1... keypoint1_angle2, keypoint2_angle2... keypoint1_angle3, keypoint1_angle3... ] ?

2 How did you generate pose into SMPL format,SPIN format , and OpenPose format? What's the order of the second dim? Is the keypoint order the same with SMPL model?

3 In pkl file: For example, dim of data in './0000/camid0_videoid2/seq0/seq0.pkl' is [48,85]. What's the order of dim 1? Is it ordered by time order or shuffled?

opened by yan811 2
GREW pretreatment `to_pickle` has size 0

I'm trying to run GREW pretreatment code but it generates no GREW-pkl folder at the end of the process. I debugged myself and checked if the --dataset flag is set properly and the to_pickle list size before saving the pickle file. The flag is well set but the size of the list is always 0.

I downloaded the GREW dataset from the link you guys sent me and made de GREW-rearranged folder using the code provided. I'll keep investigating what is causing such an error and if I find I'll set a fixing PR.

opened by gosiqueira 1
About the pose data

Can you make a detailed description of the pose data? This is the path of one frame pose and the corresponding content of the txt file Gait3D/2D_Poses/0000/camid9_videoid2/seq0/human_crop_f17279.txt '311,438,89.201164,62.87694,0.57074964,89.201164,54.322254,0.47146344,84.92382,62.87694,0.63443935,42.150383....' I have 3 questions. Q1: what does 'f17279' means? Q2: what does the first number (e.g. 311) in the txt file mean? Q3: which number('f17279' or '311') should I regard as a base when I order the sequence? Thank you very much!

opened by HiAleeYang 0

Releases(v0.1)

v0.1(Mar 25, 2022)

Upload checkpoint models
Source code(tar.gz)
Source code(zip)
128pixel-Baseline-180000.pt(309.06 MB)
128pixel-GaitGL-180000.pt(309.76 MB)
128pixel-GaitPart-180000.pt(13.85 MB)
128pixel-GaitSet-180000.pt(29.70 MB)
128pixel-GLN_P2-180000.pt(176.87 MB)
128pixel-SMPLGait_128pixel-180000.pt(312.63 MB)
64pixel-Baseline-180000.pt(309.06 MB)
64pixel-GaitGL-180000.pt(309.76 MB)
64pixel-GaitPart-180000.pt(13.85 MB)
64pixel-GaitSet-180000.pt(29.70 MB)
64pixel-GLN_P2-180000.pt(176.87 MB)
64pixel-SMPLGait_64pixel-180000.pt(310.35 MB)

Owner

Official repo for Gait3D dataset

GitHub Repository

Code for "Single-view robot pose and joint angle estimation via render & compare", CVPR 2021 (Oral).

Single-view robot pose and joint angle estimation via render & compare Yann Labbé, Justin Carpentier, Mathieu Aubry, Josef Sivic CVPR: Conference on C

51 Oct 14, 2022

Advancing Self-supervised Monocular Depth Learning with Sparse LiDAR

Official implementation for paper "Advancing Self-supervised Monocular Depth Learning with Sparse LiDAR"

72 Dec 09, 2022

[TIP 2021] SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction

SADRNet Paper link: SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction Requirements python

99 Dec 30, 2022

Code for "Adversarial Training for a Hybrid Approach to Aspect-Based Sentiment Analysis

HAABSAStar Code for "Adversarial Training for a Hybrid Approach to Aspect-Based Sentiment Analysis". This project builds on the code from https://gith

1 Sep 14, 2020

Static-test - A playground to play with ideas related to testing the comparability of the code

Static test playground ⚠️ The code is just an experiment. Compiles and runs on U

4 Feb 18, 2022

[ICCV 2021] Our work presents a novel neural rendering approach that can efficiently reconstruct geometric and neural radiance fields for view synthesis.

MVSNeRF Project page | Paper This repository contains a pytorch lightning implementation for the ICCV 2021 paper: MVSNeRF: Fast Generalizable Radiance

529 Dec 30, 2022

Code for the paper "Adapting Monolingual Models: Data can be Scarce when Language Similarity is High"

Wietse de Vries • Martijn Bartelds • Malvina Nissim • Martijn Wieling Adapting Monolingual Models: Data can be Scarce when Language Similarity is High

5 Aug 02, 2021

Locationinfo - A script helps the user to show network information such as ip address

Description This script helps the user to show network information such as ip ad

1 Dec 30, 2021

The missing CMake project initializer

cmake-init - The missing CMake project initializer Opinionated CMake project initializer to generate CMake projects that are FetchContent ready, separ

1k Jan 01, 2023

Stitch it in Time: GAN-Based Facial Editing of Real Videos

STIT - Stitch it in Time [Project Page] Stitch it in Time: GAN-Based Facial Edit

1.1k Jan 04, 2023

tsai is an open-source deep learning package built on top of Pytorch & fastai focused on state-of-the-art techniques for time series classification, regression and forecasting.

Time series Timeseries Deep Learning Pytorch fastai - State-of-the-art Deep Learning with Time Series and Sequences in Pytorch / fastai

2.8k Jan 08, 2023

Easy and Efficient Object Detector

EOD Easy and Efficient Object Detector EOD (Easy and Efficient Object Detection) is a general object detection model production framework. It aim on p

381 Jan 01, 2023

The Medical Detection Toolkit contains 2D + 3D implementations of prevalent object detectors such as Mask R-CNN, Retina Net, Retina U-Net, as well as a training and inference framework focused on dealing with medical images.

The Medical Detection Toolkit contains 2D + 3D implementations of prevalent object detectors such as Mask R-CNN, Retina Net, Retina U-Net, as well as a training and inference framework focused on dea

1.2k Jan 04, 2023