Part-aware Measurement for Robust Multi-View Multi-Human 3D Pose Estimation and Tracking

Last update: Oct 27, 2022

Overview

Part-aware Measurement for Robust Multi-View Multi-Human 3D Pose Estimation and Tracking

Part-Aware Measurement for Robust Multi-View Multi-Human 3D Pose Estimation and Tracking
Hau Chu, Jia-Hong Lee, Yao-Chih Lee, Ching-Hsien Hsu, Jia-Da Li, Chu-Song Chen
2021 CVPR B-AMFG Workshop

Note: It's a project of AI^2 Lab. The code will be update in here while there is a new version.

Installation

Python 3.6+
Cuda 9.0
Cudnn 7
gcc 5 & g++ 5 (for Ubuntu 18.04)

$ sudo apt install gcc-5 g++-5
$ sudo ln -s /usr/bin/gcc-6 /usr/local/bin/gcc
$ sudo ln -s /usr/bin/g++-6 /usr/local/bin/g++

Conda Env

$ conda create -n venv python=3.6
$ conda activate venv
$ conda install pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=9.0 -c pytorch
$ pip install tensorflow_gpu==1.9.0
$ pip install -r requirements.txt

$ sudo apt install git

Data preparation

Download datasets:

Campus (http://campar.in.tum.de/Chair/MultiHumanPose)
Shelf (http://campar.in.tum.de/Chair/MultiHumanPose)
CMU Panoptic (https://github.com/CMU-Perceptual-Computing-Lab/panoptic-toolbox)

Dataset's camera_parameter.pickle download

The directory tree should look like below:

${ROOT}
    |-- CatchImage
        |-- CampusSeq1
        |   |-- Camera0
        |   |-- Camera1
        |   |-- Camera2
        |   |-- camera_parameter.pickle
        |   |-- actorsGT.mat
        |-- Shelf
        |   |-- Camera0
        |   |-- ...
        |   |-- Camera4
        |   |-- camera_parameter.pickle
        |   |-- actorsGT.mat
        |-- Panoptic
        |   |-- 160906_pizza1
            |   |-- 00_03 # hdImgs folder of 03 camera
            |   |-- 00_06 # hdImgs folder of 06 camera
            |   |-- ...
            |   |-- camera_parameter.pickle
            |   |-- hdPose_stage1_coco19
            |-- ...
    |-- src

Backend Models

Backend models, which is not our works, are released codes from others. We only did some small modifications to fit the format of our input/output. Put models in {ROOT}/src/backend

YOLOv3
- origin: https://github.com/ZQPei/deep_sort_pytorch
- modification
HRNet
- origin: https://github.com/stefanopini/simple-HRNet
- modification

Run Codes

Demo

$cd src
python -W ignore testmodel.py --dataset CampusSeq1 # For Campus
python -W ignore testmodel.py --dataset Shelf # For Shelf
python -W ignore testmodel.py --dataset Panoptic # For Panoptic (sub-dataset can be modified in config)

Evaluation

$cd src
python -W ignore evalmodel.py --dataset CampusSeq1 
python -W ignore evalmodel.py --dataset Shelf

Campus PCP Score

Bone Group	Actor 0	Actor 1	Actor 2	Average
Head	100.00	100.00	100.00	100.00
Torso	100.00	100.00	100.00	100.00
Upper arms	98.98	100.00	100.00	99.66
Lower arms	92.86	68.78	91.30	84.31
Upper legs	100.00	100.00	100.00	100.00
Lower legs	100.00	100.00	100.00	100.00
Total	98.37	93.76	98.26	96.79

Shelf PCP Score

Bone Group	Actor 0	Actor 1	Actor 2	Average
Head	94.98	100.00	91.30	95.43
Torso	100.00	100.00	100.00	100.00
Upper arms	100.00	100.00	96.27	98.76
Lower arms	98.21	77.03	96.27	90.50
Upper legs	100.00	100.00	100.00	100.00
Lower legs	100.00	100.00	100.00	100.00
Total	99.14	95.41	97.64	97.39

Citation

@InProceedings{Chu_2021_CVPR,
    author    = {Chu, Hau and Lee, Jia-Hong and Lee, Yao-Chih and Hsu, Ching-Hsien and Li, Jia-Da and Chen, Chu-Song},
    title     = {Part-Aware Measurement for Robust Multi-View Multi-Human 3D Pose Estimation and Tracking},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2021},
    pages     = {1472-1481}
}

Part-aware Measurement for Robust Multi-View Multi-Human 3D Pose Estimation and Tracking

Related tags

Overview

Part-aware Measurement for Robust Multi-View Multi-Human 3D Pose Estimation and Tracking

Installation

Data preparation

Backend Models

Run Codes

Demo

Evaluation

Campus PCP Score

Shelf PCP Score

Citation

Owner

CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation

Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning (ICLR 2021)

MusicYOLO framework uses the object detection model, YOLOx, to locate notes in the spectrogram.

Detection of drones using their thermal signatures from thermal camera through YOLO-V3 based CNN with modifications to encapsulate drone motion

"Segmenter: Transformer for Semantic Segmentation" reproduced via mmsegmentation

Few-shot Neural Architecture Search

This is a Pytorch implementation of paper: DropEdge: Towards Deep Graph Convolutional Networks on Node Classification

Fluency ENhanced Sentence-bert Evaluation (FENSE), metric for audio caption evaluation. And Benchmark dataset AudioCaps-Eval, Clotho-Eval.

v objective diffusion inference code for JAX.

An Unbiased Learning To Rank Algorithms (ULTRA) toolbox

I explore rock vs. mine prediction using a SONAR dataset

Meta-TTS: Meta-Learning for Few-shot SpeakerAdaptive Text-to-Speech

Deep learning with dynamic computation graphs in TensorFlow

AFLFast (extends AFL with Power Schedules)

Code for "Layered Neural Rendering for Retiming People in Video."

A Pytorch implementation of CVPR 2021 paper "RSG: A Simple but Effective Module for Learning Imbalanced Datasets"

Customised to detect objects automatically by a given model file(onnx)

A Pytorch loader for MVTecAD dataset.

An open-source outlier detection package by Getcontact Data Team

Differential rendering based motion capture blender project.