CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

Overview

SMPLify-XMC

This repo is part of our project: On Self-Contact and Human Pose.
[Project Page] [Paper] [MPI Project Page]

Teaser SMPLify-XMC

License

Software Copyright License for non-commercial scientific research purposes. Please read carefully the following terms and conditions and any accompanying documentation before you download and/or use the TUCH data and software, (the "Data & Software"), including 3D meshes, images, videos, textures, software, scripts, and animations. By downloading and/or using the Data & Software (including downloading, cloning, installing, and any other use of the corresponding github repository), you acknowledge that you have read these terms and conditions, understand them, and agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use the Data & Software. Any infringement of the terms of this agreement will automatically terminate your rights under this License.

Description and Demo

SMPLify-XMC adapts SMPLify-X to fit SMPL-X model to Mimic The Pose (MTP) data. To run SMPLify-XMC you need

  • an image of a person mimicking a presented pose
  • the presented pose parameters
  • the person's gender, height and weight
  • the OpenPose keypoints.

The code has been tested with Python 3.6.9, CUDA 10.1, CuDNN 7.5 and PyTorch 1.8.1 on Ubuntu 18.04.

Installation

1) Clone this repo

git clone [email protected]:muelea/smplify-xmc.git
cd smplify-xmc

2) Download body model

Download them SMPL-X body model from https://smpl-x.is.tue.mpg.de and save it in MODEL_FOLDER. You can replace model_folder: MODEL_FOLDER in the config file configs/fit_smplx_singleview.yaml or use an environment variable.

3) Download essentials

Download essentials from here and unpack to ESSENTIALS_DIR. Then create symlinks between the essentials and this repo:

ln -s $ESSENTIALS_DIR/smplify-xmc-essentials data/essentials

4) Create python virtual environment

python3 -m venv $YOUR_VENV_DIR/smplify-xmc
source $YOUR_VENV_DIR/smplify-xmc/bin/activate

5) Install requirements

pip install -r requirements.txt

6) Get dependencies

Clone self-contact repo, e.g. to YOUR_PYTHON_PACKAGE_DIR. Then use pip to install the package. Then you can import the self-contact functions from any place in your system. (make sure your venv is activated).

cd $YOUR_PYTHON_PACKAGE_DIR
git clone [email protected]:muelea/selfcontact.git
cd selfcontact
rm -r .git
pip install .
cd ..

Demo using our example data

You can find our example dataset in this repo under data/example_input. The following command will automatically save parameters, mesh, and image under output_dir:

python main_singleview.py --config configs/fit_smplx_singleview.yaml \
--dataset mtp_demo \
--input_base_dir data/example_input/singleview/subject1 \
--input_dir_poses data/example_input/presented_poses \
--output_dir data/example_output/singleview/subject1 \
--model_folder $MODELS_FOLDER

Process the MTP dataset:

Download MTP data from the TUCH website: https://tuch.is.tue.mpg.de and save the data in DS_DIR. You should now see a folder named $DS_DIR/mtp.

Read MTP data: python lib/dataextra/preprocess_mtp_mturk_dataset.py --ds_dir=$DS_DIR/mtp

Process the first item: python main_singleview.py --config configs/fit_smplx_singleview_mtp_dataset.yaml --db_file data/dbs/mtp_mturk.npz --output_dir data/example_output/mtp/ --model_folder=$MODEL_FOLDER --cluster_bs=1 --ds_start_idx=0

Process your own data:

Follow the structure of the example data in data/example_input. Create a folder PP_FOLDER for the presented poses:

PP_FOLDER
  ----pose_name1.pkl
  ----pose_name2.pkl

The pickle file should contain a dictionary with the pose parameters and the vertices. If you include the vertices ('v'), the vertices in contact will be computed automatically.

data = {
  'body_pose': ..
  'right_hand_pose': ..
  'left_hand_pose': ..
  'global_orient': ..
  'v': .. #vertices

}

Then create a folder MI_FOLDER for the mimicked images, following the structure below. Compute the keypoints for each image from OpenPose. The meta file should contain the gender, height and weight of the subject mimicking the pose.

MI_FOLDER
  ----subject_name1
    ----images
      ----pose_name1.png
      ----pose_name2.png
    ----keypoints
      ----pose_name1.json
      ----pose_name2.json
    ----meta.yaml

Finally, run the fitting code:

python main_singleview.py --config configs/fit_smplx_singleview.yaml \
--input_base_dir $MI_FOLDER/subject_name1 \
--input_dir_poses $PP_FOLDER \
--output_dir data/example_output/subject_name1

Citation

@inproceedings{Mueller:CVPR:2021,
  title = {On Self-Contact and Human Pose},
  author = {M{\"u}ller, Lea and Osman, Ahmed A. A. and Tang, Siyu and Huang, Chun-Hao P. and Black, Michael J.},
  booktitle = {Proceedings IEEE/CVF Conf.~on Computer Vision and Pattern Recogßnition (CVPR)},
  month = jun,
  year = {2021},
  doi = {},
  month_numeric = {6}
}

Acknowledgement

We thank Vassilis Choutas and Georgios Pavlakos for publishing the SMPLify-X code: https://github.com/vchoutas/smplify-x. This has allowed us to build our code on top of it and continue to use important features, such as the prior or optimization. Again, special thanks to Vassilis Choutas for his implementation of the generalized winding numbers and the measurements code. We also thank our data capture and admin team for their help with the extensive data collection on Mechanical Turk and in the Capture Hall. Many thanks to all subjects who contributed to this dataset in the scanner and on the Internet. Thanks to all PS members who proofread the script and did not understand it and the reviewers, who helped improving during the rebuttal. Lea Mueller and Ahmed A. A. Osman thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting them. We thank the wonderful PS department for their questions and support.

Contact

For questions, please contact [email protected]

For commercial licensing (and all related questions for business applications), please contact [email protected].

Owner
Lea Müller
PhD student in the Perceiving Systems Department at the Max Planck Institute for Intelligent Systems in Tübingen, Germany.
Lea Müller
The codes reproduce the figures and statistics in the paper, "Controlling for multiple covariates," by Mark Tygert.

The accompanying codes reproduce all figures and statistics presented in "Controlling for multiple covariates" by Mark Tygert. This repository also pr

Meta Research 1 Dec 02, 2021
Code for the paper One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation, CVPR 2021.

One Thing One Click One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation (CVPR2021) Code for the paper One Thi

44 Dec 12, 2022
A modified version of DeepMind's Alphafold2 to divide CPU part (MSA and template searching) and GPU part (prediction model)

ParallelFold Author: Bozitao Zhong This is a modified version of DeepMind's Alphafold2 to divide CPU part (MSA and template searching) and GPU part (p

Bozitao Zhong 77 Dec 22, 2022
This is an official implementation for "Self-Supervised Learning with Swin Transformers".

Self-Supervised Learning with Vision Transformers By Zhenda Xie*, Yutong Lin*, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao and Han Hu This repo is the

Swin Transformer 529 Jan 02, 2023
Official PyTorch implementation of "Improving Face Recognition with Large AgeGaps by Learning to Distinguish Children" (BMVC 2021)

Inter-Prototype (BMVC 2021): Official Project Webpage This repository provides the official PyTorch implementation of the following paper: Improving F

Jungsoo Lee 16 Jun 30, 2022
Conservative Q Learning for Offline Reinforcement Reinforcement Learning in JAX

CQL-JAX This repository implements Conservative Q Learning for Offline Reinforcement Reinforcement Learning in JAX (FLAX). Implementation is built on

Karush Suri 8 Nov 07, 2022
How to Predict Stock Prices Easily Demo

How-to-Predict-Stock-Prices-Easily-Demo How to Predict Stock Prices Easily - Intro to Deep Learning #7 by Siraj Raval on Youtube ##Overview This is th

Siraj Raval 752 Nov 16, 2022
Official repository of the paper 'Essentials for Class Incremental Learning'

Essentials for Class Incremental Learning Official repository of the paper 'Essentials for Class Incremental Learning' This Pytorch repository contain

33 Nov 27, 2022
A 3D Dense mapping backend library of SLAM based on taichi-Lang designed for the aerial swarm.

TaichiSLAM This project is a 3D Dense mapping backend library of SLAM based Taichi-Lang, designed for the aerial swarm. Intro Taichi is an efficient d

XuHao 230 Dec 19, 2022
Repository for the electrical and ICT benchmark model developed in the ERIGrid 2.0 project.

Benchmark Model Electrical and ICT System This repository contains the documentation, code, and models for the electrical and ICT benchmark model deve

ERIGrid 2.0 1 Nov 29, 2021
Official code for Spoken ObjectNet: A Bias-Controlled Spoken Caption Dataset

Official code for our Interspeech 2021 - Spoken ObjectNet: A Bias-Controlled Spoken Caption Dataset [1]*. Visually-grounded spoken language datasets c

Ian Palmer 3 Jan 26, 2022
Code for the paper “The Peril of Popular Deep Learning Uncertainty Estimation Methods”

Uncertainty Estimation Methods Code for the paper “The Peril of Popular Deep Learning Uncertainty Estimation Methods” Reference If you use this code,

EPFL Machine Learning and Optimization Laboratory 4 Apr 05, 2022
SoK: Vehicle Orientation Representations for Deep Rotation Estimation

SoK: Vehicle Orientation Representations for Deep Rotation Estimation Raymond H. Tu, Siyuan Peng, Valdimir Leung, Richard Gao, Jerry Lan This is the o

FIRE Capital One Machine Learning of the University of Maryland 12 Oct 07, 2022
For the paper entitled ''A Case Study and Qualitative Analysis of Simple Cross-Lingual Opinion Mining''

Summary This is the source code for the paper "A Case Study and Qualitative Analysis of Simple Cross-Lingual Opinion Mining", which was accepted as fu

1 Nov 10, 2021
RodoSol-ALPR Dataset

RodoSol-ALPR Dataset This dataset, called RodoSol-ALPR dataset, contains 20,000 images captured by static cameras located at pay tolls owned by the Ro

Rayson Laroca 45 Dec 15, 2022
Pytorch implementation of Nueral Style transfer

Nueral Style Transfer Pytorch implementation of Nueral style transfer algorithm , it is used to apply artistic styles to content images . Content is t

Abhinav 9 Oct 15, 2022
Team Enigma at ArgMining 2021 Shared Task: Leveraging Pretrained Language Models for Key Point Matching

Team Enigma at ArgMining 2021 Shared Task: Leveraging Pretrained Language Models for Key Point Matching This is our attempt of the shared task on Quan

Manav Nitin Kapadnis 12 Jul 08, 2022
Organseg dags - The repository contains the codebase for multi-organ segmentation with directed acyclic graphs (DAGs) in CT.

Organseg dags - The repository contains the codebase for multi-organ segmentation with directed acyclic graphs (DAGs) in CT.

yzf 1 Jun 12, 2022
Facestar dataset. High quality audio-visual recordings of human conversational speech.

Facestar Dataset Description Existing audio-visual datasets for human speech are either captured in a clean, controlled environment but contain only a

Meta Research 87 Dec 21, 2022
Image-Scaling Attacks and Defenses

Image-Scaling Attacks & Defenses This repository belongs to our publication: Erwin Quiring, David Klein, Daniel Arp, Martin Johns and Konrad Rieck. Ad

Erwin Quiring 163 Nov 21, 2022