Implementation of the ICCV'21 paper Temporally-Coherent Surface Reconstruction via Metric-Consistent Atlases

Overview

Temporally-Coherent Surface Reconstruction via Metric-Consistent Atlases [Papers 1, 2][Project page] [Video]

The implementation of the papers

Install

The framework was tested with Python 3.8, PyTorch 1.7.0. and CUDA 11.0. The easiest way to work with the code is to create a new virtual Python environment and install the required packages.

  1. Install the virtualenvwrapper.
  2. Create a new environment and install the required packages.
mkvirtualenv --python=python3.8 tcsr
pip install -r requirements.txt
  1. Install Pytorch3d.
cd ~
curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
tar xzf 1.10.0.tar.gz
export CUB_HOME=$PWD/cub-1.10.0
pip install git+https://github.com/facebookresearch/[email protected]
  1. Get the code and prepare the environment as follows:
git clone [email protected]:bednarikjan/temporally_coherent_surface_reconstruction.git
git submodule update --init --recursive
export PYTHONPATH="{PYTHONPATH}:path/to/dir/temporally_coherent_surface_reconstruction"

Get the Data

The project was tested on 6 base datasets (and their derivatives). Each datasets has to be processed so as to generate the input point clouds for training, the GT correspondences for evauluation and other auxilliary data. To do so, please use the individual scripts in tcsr/process_datasets. For each dataset, follow these steps:

  1. Download the data (links below).
  2. Open the script <dataset_name>.py and set the input/output paths.
  3. Run the script: python <dataset_name>.py

1. ANIM

  • Download the sequences horse gallop, horse collapse, camel gallop, camel collapse, and elephant gallop.
  • Download the sequence walking cat.

2. AMA

  • Download all 10 sequences, meshes only.

3. DFAUST

4. CAPE

  • Request the access to the raw scans and download it.
  • At the time of writing the paper (September 2021) four subjects (00032, 00096, 00159, 03223) were available and used in the paper.

5. INRIA

  • Request the access to the dataset and download it.
  • At the time of writing the paper (September 2021), four subjects (s1, s2, s3, s6) were available and used in the paper.

6. CMU

Train

The provided code allows for training our proposed method (OUR) but also the other atlas based approaches Differential Surface Representation (DSR) and AtlasNet (AN). The training is configured using the *.yaml configuration scripts in tcsr/train/configs.

There are 9 sample configuration files our_<dataset_name>.yaml which train OUR on each individual dataset and 2 sample configuration files an_anim.yaml, dsr_anim.yaml which train AN and DSR respectivelly on ANIM dataset.

By default, the trainin uses the exact settings as in the paper, namely it trains for 200'000 iterations using SGD, learning rate of 0.001 and batch size of 4. This can be altered in the configuration files.

Before starting the training, follow these steps:

  • Open the source file tcsr/data/data_loader.py and set the paths to the datasets in each dataset class.
  • Open the desired training configuration *.yaml file in tcsr/train/configs/ and set the output path for the training run data in the attribute path_train_run.

Start the training usint the script tcsr/train/train.py:

python train.py --conf configs/<file_name>.yaml

By default the script saves the training progress each 2000 iterations so you can safely kill it at any point and resume the trianing later using:

python train.py --cont path/to/training_run/root_dir

Evaluate

To evaluate a trianed model on the dense correspondence prediction task, use the script tcsr/evaluate/eval_dataset.py which allows for evaluation of multiple sequences (i.e. individual training runs within one dataset) at once. Please have a look at the command line arguments in the file.

An example of how to run the evaluation for the training runs contained in the root directory train_runs_root corresponding to 2 training runs run for the sequences cat_walk and horse_gallop within ANIM dataset:

python eval_dataset.py /path/to/train_runs_root --ds anim --include_seqs cat_walk horse_gallop  

The script produces a *.csv file in train_runs_root with the 4 measured metrics (see the paper).

Visualize

There are currently two ways to visualize the predictions.

1. Tensorboard

By default, the training script saves the GT and the predicted point clouds (for a couple of random data samples) each 2000 iterations. These can be viewed within Tensorboard. Each patch is visualized with a different color. This visualization is mostly useful as a sanity check during the trianing to see that the model is converging as expected.

  • Navigate to the root directory of the trianing runs and run:
tensorboard --logdir=. --port=8008 --bind_all
  • Open your browser and navigate to http://localhost:8008/

2. Per-sequence reconstruction GIF

You can view the reconstructed surfaces as a patch-wise textured mesh as a video within a GIF file. For this purpose, use the IPython Notebook file tcsr/visualize/render_uv.ipynb and open it in jupyterlab which allows for viewing the GIF right after running the code.

The rendering parameters (such as the camera location, texturing mode, gif speed etc.) are set usin the configuration file tcsr/visualize/conf_patches.yaml. There are sample configurations for the sequence cat_walk, which can be used to write configurations for other sequences/datasets.

Before running the cells, set the variables in the second cell (paths, models, data).

Citation

@inproceedings{bednarik2021temporally_coherent,
   title = {Temporally-Coherent Surface Reconstruction via Metric-Consistent Atlases},
   author = {Bednarik, Jan and Kim, Vladimir G. and Chaudhuri, Siddhartha and Parashar, Shaifali and Salzmann, Mathieu and Fua, Pascal and Aigerman, Noam},
   booktitle = {Proceedings of IEEE International Conference on Computer Vision (ICCV)},
   year = {2021}
}

@inproceedings{bednarik2021temporally_consistent,
   title = {Temporally-Consistent Surface Reconstruction via Metrically-Consistent Atlases},
   author = {Bednarik, Jan and Aigerman, Noam and Kim, Vladimir G. and Chaudhuri, Siddhartha and Parashar, Shaifali and Salzmann, Mathieu and Fua, Pascal},
   booktitle = {arXiv},
   year = {2021}
}

Acknowledgements

This work was partially done while the main author was an intern at Adobe Research.

TODO

  • Add support for visualizing the correspondence error heatmap on the GT mesh.
  • Add support for visualizing the colorcoded correspondences on the GT mesh.
  • Add the support for generating the pre-aligned AMAa dataset using ICP.
  • Add the code for the nonrigid ICP experiments.
Diagnostic tests for linguistic capacities in language models

LM diagnostics This repository contains the diagnostic datasets and experimental code for What BERT is not: Lessons from a new suite of psycholinguist

61 Jan 02, 2023
Kaggleship: Kaggle Notebooks

Kaggleship: Kaggle Notebooks This repository contains my Kaggle notebooks. They are generally about data science, machine learning, and deep learning.

Erfan Sobhaei 1 Jan 25, 2022
Türkiye Canlı Mobese Görüntülerinde Profesyonel Nesne Takip Sistemi

Türkiye Mobese Görüntü Takip Türkiye Mobese görüntülerinde OPENCV ve Yolo ile takip sistemi Multiple Object Tracking System in Turkish Mobese with OPE

15 Dec 22, 2022
With this package, you can generate mixed-integer linear programming (MIP) models of trained artificial neural networks (ANNs) using the rectified linear unit (ReLU) activation function

With this package, you can generate mixed-integer linear programming (MIP) models of trained artificial neural networks (ANNs) using the rectified linear unit (ReLU) activation function. At the momen

ChemEngAI 40 Dec 27, 2022
Monitora la qualità della ricezione dei segnali radio nelle province siciliane.

FMap-server Monitora la qualità della ricezione dei segnali radio nelle province siciliane. Conversion data Frequency - StationName maps are stored in

Triglie 5 May 24, 2021
An implementation of Equivariant e2 convolutional kernals into a convolutional self attention network, applied to radio astronomy data.

EquivariantSelfAttention An implementation of Equivariant e2 convolutional kernals into a convolutional self attention network, applied to radio astro

2 Nov 09, 2021
DARTS-: Robustly Stepping out of Performance Collapse Without Indicators

[ICLR'21] DARTS-: Robustly Stepping out of Performance Collapse Without Indicators [openreview] Authors: Xiangxiang Chu, Xiaoxing Wang, Bo Zhang, Shun

55 Nov 01, 2022
Code Impementation for "Mold into a Graph: Efficient Bayesian Optimization over Mixed Spaces"

Code Impementation for "Mold into a Graph: Efficient Bayesian Optimization over Mixed Spaces" This repo contains the implementation of GEBO algorithm.

Jaeyeon Ahn 2 Mar 22, 2022
This codebase is the official implementation of Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization (NeurIPS2021, Spotlight)

Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization This codebase is the official implementation of Test-Time Classifier A

47 Dec 28, 2022
PyTorch implementation of "Contrast to Divide: self-supervised pre-training for learning with noisy labels"

Contrast to Divide: self-supervised pre-training for learning with noisy labels This is an official implementation of "Contrast to Divide: self-superv

55 Nov 23, 2022
A Semantic Segmentation Network for Urban-Scale Building Footprint Extraction Using RGB Satellite Imagery

A Semantic Segmentation Network for Urban-Scale Building Footprint Extraction Using RGB Satellite Imagery This repository is the official implementati

Aatif Jiwani 42 Dec 08, 2022
A Python 3 package for state-of-the-art statistical dimension reduction methods

direpack: a Python 3 library for state-of-the-art statistical dimension reduction techniques This package delivers a scikit-learn compatible Python 3

Sven Serneels 32 Dec 14, 2022
CDGAN: Cyclic Discriminative Generative Adversarial Networks for Image-to-Image Transformation

CDGAN CDGAN: Cyclic Discriminative Generative Adversarial Networks for Image-to-Image Transformation CDGAN Implementation in PyTorch This is the imple

Kancharagunta Kishan Babu 6 Apr 19, 2022
Offline Reinforcement Learning with Implicit Q-Learning

Offline Reinforcement Learning with Implicit Q-Learning This repository contains the official implementation of Offline Reinforcement Learning with Im

Ilya Kostrikov 126 Jan 06, 2023
Voice of Pajlada with model and weights.

Pajlada TTS Stripped down version of ForwardTacotron (https://github.com/as-ideas/ForwardTacotron) with pretrained weights for Pajlada's (https://gith

6 Sep 03, 2021
Code accompanying "Dynamic Neural Relational Inference" from CVPR 2020

Code accompanying "Dynamic Neural Relational Inference" This codebase accompanies the paper "Dynamic Neural Relational Inference" from CVPR 2020. This

Colin Graber 48 Dec 23, 2022
Recommendation algorithms for large graphs

Fast recommendation algorithms for large graphs based on link analysis. License: Apache Software License Author: Emmanouil (Manios) Krasanakis Depende

Multimedia Knowledge and Social Analytics Lab 27 Jan 07, 2023
Exploring the Dual-task Correlation for Pose Guided Person Image Generation

Dual-task Pose Transformer Network The source code for our paper "Exploring Dual-task Correlation for Pose Guided Person Image Generation“ (CVPR2022)

63 Dec 15, 2022
Unofficial PyTorch implementation of Masked Autoencoders Are Scalable Vision Learners

Unofficial PyTorch implementation of Masked Autoencoders Are Scalable Vision Learners This repository is built upon BEiT, thanks very much! Now, we on

Zhiliang Peng 2.3k Jan 04, 2023
Official implementation of the ICCV 2021 paper "Conditional DETR for Fast Training Convergence".

The DETR approach applies the transformer encoder and decoder architecture to object detection and achieves promising performance. In this paper, we handle the critical issue, slow training convergen

281 Dec 30, 2022