GAN-based 3D human pose estimation model for 3DV'17 paper

Overview

Tensorflow implementation for 3DV 2017 conference paper "Adversarially Parameterized Optimization for 3D Human Pose Estimation".

@inproceedings{jack2017adversarially,
  title={Adversarially Parameterized Optimization for 3D Human Pose Estimation},
  author={Jack, Dominic and Maire, Frederic and Eriksson, Anders and Shirazi, Sareh},
  booktitle={3D Vision (3DV), 2017 Fifth International Conference on},
  year={2017},
  organization={IEEE}
}

Code used to generate results for the paper has been frozen and can be found in the 3dv2017 branch. Bug fixes and extensions will be applied to other branches.

Algorithm Overview

The premise of the paper is to train a GAN to simultaneously learn a parameterization of the feasible human pose space along with a feasibility loss function.

During inference, a standard off-the-shelf optimizer infers all poses from sequence almost-independently (the scale is shared between frames, which has no effect on the results (since errors are on the procruste-aligned inferences which optimize over scale) but makes the visualizations easier to interpret).

Repository Structure

Each GAN is identified by a gan_id. Hyperparameters defining the network structures and datasets from which they should be trained are specified in gan_params/gan_id.json. A couple (those with results highlighted in the paper) are provided, h3m_big, h3m_small and eva_big. Note that compared to typical neural networks, these are still tiny, so the difference in size should result in a negligible difference in training/inference time.

Similarly, each inference run is identified by an inference_id, the parameters of which are defined in inference_params/inference_id.json. including geometric transforms, visualizations and dataset reading

  • gan: provides application-specific GANs based on specifications in gan_params
  • serialization.py: i/o related functions for loading hyper-parameters/results

Scripts:

  • train.py: Trains a GAN specified by a json file in gan_params
  • gan_generator_vis.py: visualization script for a trained GAN generator
  • interactive_gan_generator_vis.ipynb: interactive jupyter/ipython notebook for visualizing a trained GAN generator
  • generate_inferences.py: Generates inferences based on parameters specified by a json file in inference_params
  • h3m_report.py/eva_report.py: reporting scripts for generated inferences.
  • vis_sequecne.py: visualization script for entire inferred sequence.

Usage

  1. Setup the external repositories: * human_pose_util
  2. Clone this repository and add the location and the parent directory(s) to your PYTHONPATH
cd path/to/parent_folder
git clone https://github.com/jackd/adversarially_parameterized_optimization.git
git clone https://github.com/jackd/human_pose_util.git
export PYTHONPATH=/path/to/parent_folder:$PYTHONPATH
cd adversarially_parameterized_optimization
  1. Define a GAN model by creating a gan_params/gan_id.json file, or select one of the existing ones.
  2. Setup the relevant dataset(s) or create your own as described in human_pose_util.
  3. Train the GAN
python train.py gan_id --max_steps=1e7

Our experiments were conducted on an NVidia K620 Quadro GPU with 2GB memory. Training runs at ~600 batches per second with a batch size of 128. For 10 million steps (likely excessive) this takes around 4.5 hours.

View training progress and compare different runs using tensorboard:

tensorboard --logdir=models
  1. (Optional) Check your generator is behaving well by running gan_generator_vis.py model_id or interactively by running interactive_gan_generator_vis.ipynb and modifying the model_id.
  2. Define an inference specification by creating an inference_params/inference_id.json file, or select one of the defaults provided.
  3. Generate inference
python generate_inferences.py inference_id

Sequence optimization runs at ~5-10fps (speed-up compared to 1fps reported in paper due to reimplementation efficiencies rather than different ideas).

This will save results in results.hdf5 in the inference_id group. 9. See the results! * h3m_report.py or eva_report.py depending on the dataset gives qualitative results

python report.py eval_id
* `vis_sequence.py` visualizes inferences

Note that results are quite unstable with respect to GAN training. You may get considerably different quantitative results than those published in the paper, though qualitative behaviour should be similar.

Serialization

To aid with experiments with different parameter sets, model/inference parameters are saved in json for ease of parsing and human readability. To allow for extensibility, human_pose_util maintains registers for different datasets and skeletons.

See the README for details on setting up/preprocessing of datasets or implementing your own.

The scripts in this project register some default h3m/eva datasets using register_defaults. While normally fast, some data conversion is performed the first time this function is run for each dataset and requires the original datasets be available with paths defined (see below). If you only wish to experiment with one dataset -- e.g. h3m -- modify the default argument values for register_defaults, e.g. def register_defaults(h3m=True, eva=False): (or the relevant function calls).

If you implement your own datasets/skeletons, either add their registrations to the default functions, or edit the relevant scripts to register them manually.

Datasets

See human_pose_util repository for instructions for setting up datasets.

Requirements

For training/inference:

  • tensorflow 1.4
  • numpy
  • h5py For visualizations:
  • matplotlib
  • glumpy (install from source may reduce issues) For initial human 3.6m dataset transformations:
  • spacepy (for initial human 3.6m dataset conversion to hdf5)

Development

This branch will be actively maintained, updated and extended. For code used to generate results for the publication, see the 3dv2017 branch.

Contact

Please report any issues/bugs. Feature requests in this repository will largely be ignored, but will be considered if made in independent repositories.

Email contact to discuss ideas/collaborations welcome: [email protected].

Owner
Dominic Jack
Deep Learning / Cybsecurity Researcher
Dominic Jack
A benchmark dataset for emulating atmospheric radiative transfer in weather and climate models with machine learning (NeurIPS 2021 Datasets and Benchmarks Track)

ClimART - A Benchmark Dataset for Emulating Atmospheric Radiative Transfer in Weather and Climate Models Official PyTorch Implementation Using deep le

21 Dec 31, 2022
Gems & Holiday Package Prediction

Predictive_Modelling Gems & Holiday Package Prediction This project is based on 2 cases studies : Gems Price Prediction and Holiday Package prediction

Avnika Mehta 1 Jan 27, 2022
Blind Image Super-resolution with Elaborate Degradation Modeling on Noise and Kernel

Blind Image Super-resolution with Elaborate Degradation Modeling on Noise and Kernel This repository is the official PyTorch implementation of BSRDM w

Zongsheng Yue 69 Jan 05, 2023
YOLOv4-v3 Training Automation API for Linux

This repository allows you to get started with training a state-of-the-art Deep Learning model with little to no configuration needed! You provide your labeled dataset or label your dataset using our

BMW TechOffice MUNICH 626 Dec 31, 2022
ICCV2021 Paper: AutoShape: Real-Time Shape-Aware Monocular 3D Object Detection

ICCV2021 Paper: AutoShape: Real-Time Shape-Aware Monocular 3D Object Detection

Zongdai 107 Dec 20, 2022
Bayesian Meta-Learning Through Variational Gaussian Processes

vmgp This is the repository of Vivek Myers and Nikhil Sardana for our CS 330 final project, Bayesian Meta-Learning Through Variational Gaussian Proces

Vivek Myers 2 Nov 17, 2022
Python版OpenCVのTracking APIのサンプルです。DaSiamRPNアルゴリズムまで対応しています。

OpenCV-Object-Tracker-Sample Python版OpenCVのTracking APIのサンプルです。   Requirement opencv-contrib-python 4.5.3.56 or later Algorithm 2021/07/16時点でOpenCVには以

KazuhitoTakahashi 36 Jan 01, 2023
This codebase is the official implementation of Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization (NeurIPS2021, Spotlight)

Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization This codebase is the official implementation of Test-Time Classifier A

47 Dec 28, 2022
PyTorch implementation of SIFT descriptor

This is an differentiable pytorch implementation of SIFT patch descriptor. It is very slow for describing one patch, but quite fast for batch. It can

Dmytro Mishkin 150 Dec 24, 2022
Language model Prompt And Query Archive

LPAQA: Language model Prompt And Query Archive This repository contains data and code for the paper How Can We Know What Language Models Know? Install

127 Dec 20, 2022
A simple algorithm for extracting tree height in sparse scene from point cloud data.

TREE HEIGHT EXTRACTION IN SPARSE SCENES BASED ON UAV REMOTE SENSING This is the offical python implementation of the paper "Tree Height Extraction in

6 Oct 28, 2022
A object detecting neural network powered by the yolo architecture and leveraging the PyTorch framework and associated libraries.

Yolo-Powered-Detector A object detecting neural network powered by the yolo architecture and leveraging the PyTorch framework and associated libraries

Luke Wilson 1 Dec 03, 2021
Layered Neural Atlases for Consistent Video Editing

Layered Neural Atlases for Consistent Video Editing Project Page | Paper This repository contains an implementation for the SIGGRAPH Asia 2021 paper L

Yoni Kasten 353 Dec 27, 2022
E2VID_ROS - E2VID_ROS: E2VID to a real-time system

E2VID_ROS Introduce We extend E2VID to a real-time system. Because Python ROS ca

Robin Shaun 7 Apr 17, 2022
Out-of-boundary View Synthesis towards Full-frame Video Stabilization

Out-of-boundary View Synthesis towards Full-frame Video Stabilization Introduction | Update | Results Demo | Introduction This repository contains the

25 Oct 10, 2022
This is the official PyTorch implementation of our paper: "Artistic Style Transfer with Internal-external Learning and Contrastive Learning".

Artistic Style Transfer with Internal-external Learning and Contrastive Learning This is the official PyTorch implementation of our paper: "Artistic S

51 Dec 20, 2022
Official repository of the paper 'Essentials for Class Incremental Learning'

Essentials for Class Incremental Learning Official repository of the paper 'Essentials for Class Incremental Learning' This Pytorch repository contain

33 Nov 27, 2022
This is a simple face recognition mini project that was completed by a team of 3 members in 1 week's time

PeekingDuckling 1. Description This is an implementation of facial identification algorithm to detect and identify the faces of the 3 team members Cla

Eric Kwok 2 Jan 25, 2022
Human Dynamics from Monocular Video with Dynamic Camera Movements

Human Dynamics from Monocular Video with Dynamic Camera Movements Ri Yu, Hwangpil Park and Jehee Lee Seoul National University ACM Transactions on Gra

215 Jan 01, 2023
an implementation of Video Frame Interpolation via Adaptive Separable Convolution using PyTorch

This work has now been superseded by: https://github.com/sniklaus/revisiting-sepconv sepconv-slomo This is a reference implementation of Video Frame I

Simon Niklaus 985 Jan 08, 2023