PyTorch reimplementation of hand-biomechanical-constraints (ECCV2020)

Overview

Hand Biomechanical Constraints Pytorch

Unofficial PyTorch reimplementation of Hand-Biomechanical-Constraints (ECCV2020).

This project reimplement following components :

  1. 3 kinds of biomechanical soft constraints
  2. integrate BMC into training procedure (PyTorch version)

Usage

  • Retrieve the code
git clone https://github.com/MengHao666/Hand-BMC-pytorch
cd Hand-BMC-pytorch
  • Create and activate the virtual environment with python dependencies
conda env create --file=environment.yml
conda activate bmc

Download data

Download 3D joint location data joints.zip Google Drive or Baidu Pan (2pip), and . These statistics are from following datasets:

Note the data from these datasets under their own licenses.

Calculate BMC

BMC

Run the code

python calculate_bmc.py

You will get

  • bone_len_max.npy bone_len_min.npy for bone length limits
  • curvatures_max.npy curvatures_min.npy for Root bones' curvatures
  • PHI_max.npy PHI_min.npy for Root bones' angular distance
  • joint_angles.npy for Joint angles

And if u want to check the coordinate system, run the code

cd utils
python calculate_joint_angles.py
  • red ,green, blue arrows refer to X,Y,Z of local coordinate system respectively;
  • dark arrows refer to bones;
  • pink arrows refer to bone projection into X-Z plane of local coordinate system;
One view Another view

Run the code

python calculate_convex_hull.py

You will get CONVEX_HULLS.npy, i.e. convex hulls to encircle the anatomically plausible joint angles.

And you will also see every convex hull like following figure:

BMC

  • "Bone PIP" means the bone from MCP joint to PIP joint in thumb
  • flexion and abduction is two kinds of angle describing joint rotation
  • "ori_convex_hull" means the original convex hull calculated from all joint angle points
  • "rdp_convex_hull" means convex hull simplified by the Ramer-Douglas-Peucker algorithm, a polygon simplification algorithm
  • "del_convex_hull" means convex hull further simplified by a greedy algorithm
  • "rectangle" means the minimal rectangle to surround all joint angle points

Run the code

python plot.py

You will see all the convex hulls

BMC

Integrate BMC into training (PyTorch version)

Run the code

python weakloss.py

Experiment results

To check influence of BMC, instead of reimplementing the network of origin paper, I integrate BMC into my own project,

Train and evaluation curve

(AUC means 3D PCK, and ACC_HM means 2D PCK) teaser

3D PCK AUC Diffenence

Dataset DetNet DetNet+BMC
RHD 0.9339 0.9364
STB 0.8744 0.8778
DO 0.9378 0.9475
EO 0.9270 0.9182

Note

  • Adjusting training parameters carefully, longer training time might further boost accuracy.
  • As BMC is a weakly supervised method, it may only make predictions more physically plausible,but cannot boost AUC performance strongly when strong supervision is used.

Limitation

  • Due to time limitation, I didn't reimplement the network and experiments of original paper.
  • There is a little difference between original paper and my reimplementation. But most of them match.

Citation

This is the unofficial pytorch reimplementation of the paper "Weakly supervised 3d hand pose estimation via biomechanical constraints (ECCV 2020).

If you find the project helpful, please star this project and cite them:

@article{spurr2020weakly,
  title={Weakly supervised 3d hand pose estimation via biomechanical constraints},
  author={Spurr, Adrian and Iqbal, Umar and Molchanov, Pavlo and Hilliges, Otmar and Kautz, Jan},
  journal={arXiv preprint arXiv:2003.09282},
  volume={8},
  year={2020},
  publisher={Springer}
}
Owner
Hao Meng
Master student at Beihang University , mainly interested in hand pose estimation.
Hao Meng
pybaum provides tools to work with pytrees which is a concept burrowed from JAX.

pybaum provides tools to work with pytrees which is a concept burrowed from JAX.

Open Source Economics 9 May 11, 2022
Inverse Optimal Control Adapted to the Noise Characteristics of the Human Sensorimotor System

Inverse Optimal Control Adapted to the Noise Characteristics of the Human Sensorimotor System This repository contains code for the paper Schultheis,

2 Oct 28, 2022
Scenic: A Jax Library for Computer Vision and Beyond

Scenic Scenic is a codebase with a focus on research around attention-based models for computer vision. Scenic has been successfully used to develop c

Google Research 1.6k Dec 27, 2022
Pretraining on Dynamic Graph Neural Networks

Pretraining on Dynamic Graph Neural Networks Our article is PT-DGNN and the code is modified based on GPT-GNN Requirements python 3.6 Ubuntu 18.04.5 L

7 Dec 17, 2022
Memory Defense: More Robust Classificationvia a Memory-Masking Autoencoder

Memory Defense: More Robust Classificationvia a Memory-Masking Autoencoder Authors: - Eashan Adhikarla - Dan Luo - Dr. Brian D. Davison Abstract Many

Eashan Adhikarla 4 Dec 25, 2022
PyTorch Language Model for 1-Billion Word (LM1B / GBW) Dataset

PyTorch Large-Scale Language Model A Large-Scale PyTorch Language Model trained on the 1-Billion Word (LM1B) / (GBW) dataset Latest Results 39.98 Perp

Ryan Spring 114 Nov 04, 2022
The 1st Place Solution of the Facebook AI Image Similarity Challenge (ISC21) : Descriptor Track.

ISC21-Descriptor-Track-1st The 1st Place Solution of the Facebook AI Image Similarity Challenge (ISC21) : Descriptor Track. You can check our solution

lyakaap 75 Jan 08, 2023
Forecasting with Gradient Boosted Time Series Decomposition

ThymeBoost ThymeBoost combines time series decomposition with gradient boosting to provide a flexible mix-and-match time series framework for spicy fo

131 Jan 08, 2023
This is a repository for a semantic segmentation inference API using the OpenVINO toolkit

BMW-IntelOpenVINO-Segmentation-Inference-API This is a repository for a semantic segmentation inference API using the OpenVINO toolkit. It's supported

BMW TechOffice MUNICH 34 Nov 24, 2022
AdelaiDet is an open source toolbox for multiple instance-level detection and recognition tasks.

AdelaiDet is an open source toolbox for multiple instance-level detection and recognition tasks.

Adelaide Intelligent Machines (AIM) Group 3k Jan 02, 2023
MCMC samplers for Bayesian estimation in Python, including Metropolis-Hastings, NUTS, and Slice

Sampyl May 29, 2018: version 0.3 Sampyl is a package for sampling from probability distributions using MCMC methods. Similar to PyMC3 using theano to

Mat Leonard 304 Dec 25, 2022
ManipulaTHOR, a framework that facilitates visual manipulation of objects using a robotic arm

ManipulaTHOR: A Framework for Visual Object Manipulation Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli VanderBilt, Luca Weihs, Eric Kolve, Aniruddha

AI2 65 Dec 30, 2022
Camera-caps - Examine the camera capabilities for V4l2 cameras

camera-caps This is a graphical user interface over the v4l2-ctl command line to

Jetsonhacks 25 Dec 26, 2022
CLIP + VQGAN / PixelDraw

clipit Yet Another VQGAN-CLIP Codebase This started as a fork of @nerdyrodent's VQGAN-CLIP code which was based on the notebooks of @RiversWithWings a

dribnet 276 Dec 12, 2022
Multi-Stage Spatial-Temporal Convolutional Neural Network (MS-GCN)

Multi-Stage Spatial-Temporal Convolutional Neural Network (MS-GCN) This code implements the skeleton-based action segmentation MS-GCN model from Autom

Benjamin Filtjens 8 Nov 29, 2022
NPBG++: Accelerating Neural Point-Based Graphics

[CVPR 2022] NPBG++: Accelerating Neural Point-Based Graphics Project Page | Paper This repository contains the official Python implementation of the p

Ruslan Rakhimov 57 Dec 03, 2022
chainladder - Property and Casualty Loss Reserving in Python

chainladder (python) chainladder - Property and Casualty Loss Reserving in Python This package gets inspiration from the popular R ChainLadder package

Casualty Actuarial Society 130 Dec 07, 2022
Implements the training, testing and editing tools for "Pluralistic Image Completion"

Pluralistic Image Completion ArXiv | Project Page | Online Demo | Video(demo) This repository implements the training, testing and editing tools for "

Chuanxia Zheng 615 Dec 08, 2022
AOT-GAN for High-Resolution Image Inpainting (codebase for image inpainting)

AOT-GAN for High-Resolution Image Inpainting Arxiv Paper | AOT-GAN: Aggregated Contextual Transformations for High-Resolution Image Inpainting Yanhong

Multimedia Research 214 Jan 03, 2023
Official implementation of CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification

CrossViT This repository is the official implementation of CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification. ArXiv If

International Business Machines 168 Dec 29, 2022