PyTorch implementation of hand mesh reconstruction described in CMR and MobRecon.

Overview

Hand Mesh Reconstruction

Introduction

This repo is the PyTorch implementation of hand mesh reconstruction described in CMR and MobRecon.

Update

  • 2021-12.7, Add MobRecon demo.
  • 2021-6-10, Add Human3.6M dataset.
  • 2021-5-20, Add CMR-G model.

Features

  • SpiralNet++
  • Sub-pose aggregation
  • Adaptive 2D-1D registration for mesh-image alignment
  • DenseStack for 2D encoding
  • Feature lifting with MapReg and PVL
  • DSConv as an efficient mesh operator
  • MobRecon training with consistency learning and complement data

Install

  • Environment

    conda create -n handmesh python=3.6
    conda activate handmesh
    
  • Please follow official suggestions to install pytorch and torchvision. We use pytorch=1.7.1, torchvision=0.8.2

  • Requirements

    pip install -r requirements.txt
    

    If you have difficulty in installing torch_sparse etc., please use whl file from here.

  • MPI-IS Mesh: We suggest to install this library from the source

  • Download the files you need from Google drive.

Run a demo

  • Prepare pre-trained models as

    out/Human36M/cmr_g/checkpoints/cmr_g_res18_human36m.pt
    out/FreiHAND/cmr_g/checkpoints/cmr_g_res18_moredata.pt
    out/FreiHAND/cmr_sg/checkpoints/cmr_sg_res18_freihand.pt
    out/FreiHAND/cmr_pg/checkpoints/cmr_pg_res18_freihand.pt  
    out/FreiHAND/mobrecon/checkpoints/mobrecon_densestack_dsconv.pt  
    
  • Run

    ./scripts/demo_cmr.sh
    ./scripts/demo_mobrecon.sh
    

    The prediction results will be saved in output directory, e.g., out/FreiHAND/mobrecon/demo.

  • Explaination of the output

    • In an JPEG file (e.g., 000_plot.jpg), we show silhouette, 2D pose, projection of mesh, camera-space mesh and pose
    • As for camera-space information, we use a red rectangle to indicate the camera position, or the image plane. The unit is meter.
    • If you run the demo, you can also obtain a PLY file (e.g., 000_mesh.ply).
      • This file is a 3D model of the hand.
      • You can open it with corresponding software (e.g., Preview in Mac).
      • Here, you can get more 3D details through rotation and zoom in.

Dataset

FreiHAND

  • Please download FreiHAND dataset from this link, and create a soft link in data, i.e., data/FreiHAND.
  • Download mesh GT file freihand_train_mesh.zip, and unzip it under data/FreiHAND/training

Human3.6M

  • The official data is now not avaliable. Please follow I2L repo to download it.
  • Download silhouette GT file h36m_mask.zip, and unzip it under data/Human36M.

Data dir

${ROOT}  
|-- data  
|   |-- FreiHAND
|   |   |-- training
|   |   |   |-- rgb
|   |   |   |-- mask
|   |   |   |-- mesh
|   |   |-- evaluation
|   |   |   |-- rgb
|   |   |-- evaluation_K.json
|   |   |-- evaluation_scals.json
|   |   |-- training_K.json
|   |   |-- training_mano.json
|   |   |-- training_xyz.json
|   |-- Human3.6M
|   |   |-- images
|   |   |-- mask
|   |   |-- annotations

Evaluation

FreiHAND

./scripts/eval_cmr_freihand.sh
./scripts/eval_mobrecon_freihand.sh
  • JSON file will be saved as out/FreiHAND/cmr_sg/cmr_sg.josn. You can submmit this file to the official server for evaluation.

Human3.6M

./scripts/eval_cmr_human36m.sh

Performance on PA-MPJPE (mm)

We re-produce the following results after code re-organization.

Model / Dataset FreiHAND Human3.6M (w/o COCO)
CMR-G-ResNet18 7.6 -
CMR-SG-ResNet18 7.5 -
CMR-PG-ResNet18 7.5 50.0
MobRecon-DenseStack 6.9 -

Training

./scripts/train_cmr_freihand.sh
./scripts/train_cmr_human36m.sh

Reference

@inproceedings{bib:CMR,
  title={Camera-Space Hand Mesh Recovery via Semantic Aggregationand Adaptive 2D-1D Registration},
  author={Chen, Xingyu and Liu, Yufeng and Ma, Chongyang and Chang, Jianlong and Wang, Huayan and Chen, Tian and Guo, Xiaoyan and Wan, Pengfei and Zheng, Wen},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2021}
}
@article{bib:MobRecon,
  title={MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image},
  author={Chen, Xingyu and Liu, Yufeng and Dong Yajiao and Zhang, Xiong and Ma, Chongyang and Xiong, Yanmin and Zhang, Yuan and Guo, Xiaoyan},
  journal={arXiv:2112.02753},
  year={2021}
}
}

Acknowledgement

Our implementation of SpiralConv is based on spiralnet_plus.

Owner
Xingyu Chen
Xingyu Chen
OpenMMLab Image and Video Editing Toolbox

Introduction MMEditing is an open source image and video editing toolbox based on PyTorch. It is a part of the OpenMMLab project. The master branch wo

OpenMMLab 3.9k Jan 04, 2023
A fast implementation of bss_eval metrics for blind source separation

fast_bss_eval Do you have a zillion BSS audio files to process and it is taking days ? Is your simulation never ending ? Fear no more! fast_bss_eval i

Robin Scheibler 99 Dec 13, 2022
Space robot - (Course Project) Using the space robot to capture the target satellite that is disabled and spinning, then stabilize and fix it up

Space robot - (Course Project) Using the space robot to capture the target satellite that is disabled and spinning, then stabilize and fix it up

Mingrui Yu 3 Jan 07, 2022
Unsupervised Representation Learning via Neural Activation Coding

Neural Activation Coding This repository contains the code for the paper "Unsupervised Representation Learning via Neural Activation Coding" published

yookoon park 5 May 26, 2022
Implementation of the paper Scalable Intervention Target Estimation in Linear Models (NeurIPS 2021), and the code to generate simulation results.

Scalable Intervention Target Estimation in Linear Models Implementation of the paper Scalable Intervention Target Estimation in Linear Models (NeurIPS

0 Oct 25, 2021
The official code of "SCROLLS: Standardized CompaRison Over Long Language Sequences".

SCROLLS This repository contains the official code of the paper: "SCROLLS: Standardized CompaRison Over Long Language Sequences". Links Official Websi

TAU NLP Group 39 Dec 23, 2022
The Habitat-Matterport 3D Research Dataset - the largest-ever dataset of 3D indoor spaces.

Habitat-Matterport 3D Dataset (HM3D) The Habitat-Matterport 3D Research Dataset is the largest-ever dataset of 3D indoor spaces. It consists of 1,000

Meta Research 62 Dec 27, 2022
Code of paper: "DropAttack: A Masked Weight Adversarial Training Method to Improve Generalization of Neural Networks"

DropAttack: A Masked Weight Adversarial Training Method to Improve Generalization of Neural Networks Abstract: Adversarial training has been proven to

倪仕文 (Shiwen Ni) 58 Nov 10, 2022
A framework for analyzing computer vision models with simulated data

3DB: A framework for analyzing computer vision models with simulated data Paper Quickstart guide Blog post Installation Follow instructions on: https:

3DB 112 Jan 01, 2023
An official reimplementation of the method described in the INTERSPEECH 2021 paper - Speech Resynthesis from Discrete Disentangled Self-Supervised Representations.

Speech Resynthesis from Discrete Disentangled Self-Supervised Representations Implementation of the method described in the Speech Resynthesis from Di

Facebook Research 253 Jan 06, 2023
Easy genetic ancestry predictions in Python

ezancestry Easily visualize your direct-to-consumer genetics next to 2500+ samples from the 1000 genomes project. Evaluate the performance of a custom

Kevin Arvai 38 Jan 02, 2023
Hierarchical User Intent Graph Network for Multimedia Recommendation

Hierarchical User Intent Graph Network for Multimedia Recommendation This is our Pytorch implementation for the paper: Hierarchical User Intent Graph

6 Jan 05, 2023
DeiT: Data-efficient Image Transformers

DeiT: Data-efficient Image Transformers This repository contains PyTorch evaluation code, training code and pretrained models for DeiT (Data-Efficient

Facebook Research 3.2k Jan 06, 2023
Code for Neural-GIF: Neural Generalized Implicit Functions for Animating People in Clothing(ICCV21)

NeuralGIF Code for Neural-GIF: Neural Generalized Implicit Functions for Animating People in Clothing(ICCV21) We present Neural Generalized Implicit F

Garvita Tiwari 104 Nov 18, 2022
Code for the AI lab course 2021/2022 of the University of Verona

AI-Lab Code for the AI lab course 2021/2022 of the University of Verona Set-Up the environment for the curse Download Anaconda for your System. Instal

Davide Corsi 5 Oct 19, 2022
Code for CVPR2021 paper "Learning Salient Boundary Feature for Anchor-free Temporal Action Localization"

AFSD: Learning Salient Boundary Feature for Anchor-free Temporal Action Localization This is an official implementation in PyTorch of AFSD. Our paper

Tencent YouTu Research 146 Dec 24, 2022
Complete* list of autonomous driving related datasets

AD Datasets Complete* and curated list of autonomous driving related datasets Contributing Contributions are very welcome! To add or update a dataset:

Daniel Bogdoll 13 Dec 19, 2022
A comprehensive and up-to-date developer education platform for Urbit.

curriculum A comprehensive and up-to-date developer education platform for Urbit. This project organizes developer capabilities into a hierarchy of co

Sigilante 36 Oct 04, 2022
Semantic Image Synthesis with SPADE

Semantic Image Synthesis with SPADE New implementation available at imaginaire repository We have a reimplementation of the SPADE method that is more

NVIDIA Research Projects 7.3k Jan 07, 2023
Implementation detail for paper "Multi-level colonoscopy malignant tissue detection with adversarial CAC-UNet"

Multi-level-colonoscopy-malignant-tissue-detection-with-adversarial-CAC-UNet Implementation detail for our paper "Multi-level colonoscopy malignant ti

CVSM Group - email: <a href=[email protected]"> 84 Nov 22, 2022