[NeurIPS'21] Shape As Points: A Differentiable Poisson Solver

Overview

Shape As Points (SAP)

Paper | Project Page | Short Video (6 min) | Long Video (12 min)

This repository contains the implementation of the paper:

Shape As Points: A Differentiable Poisson Solver
Songyou Peng, Chiyu "Max" Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys and Andreas Geiger
NeurIPS 2021 (Oral)

If you find our code or paper useful, please consider citing

@inproceedings{Peng2021SAP,
 author    = {Peng, Songyou and Jiang, Chiyu "Max" and Liao, Yiyi and Niemeyer, Michael and Pollefeys, Marc and Geiger, Andreas},
 title     = {Shape As Points: A Differentiable Poisson Solver},
 booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
 year      = {2021}}

Installation

First you have to make sure that you have all dependencies in place. The simplest way to do so, is to use anaconda.

You can create an anaconda environment called sap using

conda env create -f environment.yaml
conda activate sap

Now, you can install PyTorch3D 0.6.0 from the official instruction as follows

pip install pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_cu102_pyt190/download.html

And install PyTorch Scatter:

conda install pytorch-scatter -c pyg

Demo - Quick Start

First, run the script to get the demo data:

bash scripts/download_demo_data.sh

Optimization-based 3D Surface Reconstruction

You can now quickly test our code on the data shown in the teaser. To this end, simply run:

python optim_hierarchy.py configs/optim_based/teaser.yaml

This script should create a folder out/demo_optim where the output meshes and the optimized oriented point clouds under different grid resolution are stored.

To visualize the optimization process on the fly, you can set o3d_show: Frue in configs/optim_based/teaser.yaml.

Learning-based 3D Surface Reconstruction

You can also test SAP on another application where we can reconstruct from unoriented point clouds with either large noises or outliers with a learned network.

For the point clouds with large noise as shown above, you can run:

python generate.py configs/learning_based/demo_large_noise.yaml

The results can been found at out/demo_shapenet_large_noise/generation/vis.

As for the point clouds with outliers, you can run:

python generate.py configs/learning_based/demo_outlier.yaml

You can find the reconstrution on out/demo_shapenet_outlier/generation/vis.

Dataset

We have different dataset for our optimization-based and learning-based settings.

Dataset for Optimization-based Reconstruction

Here we consider the following dataset:

Please cite the corresponding papers if you use the data.

You can download the processed dataset (~200 MB) by running:

bash scripts/download_optim_data.sh

Dataset for Learning-based Reconstruction

We train and evaluate on ShapeNet. You can download the processed dataset (~220 GB) by running:

bash scripts/download_shapenet.sh

After, you should have the dataset in data/shapenet_psr folder.

Alternatively, you can also preprocess the dataset yourself. To this end, you can:

Usage for Optimization-based 3D Reconstruction

For our optimization-based setting, you can consider running with a coarse-to-fine strategy:

python optim_hierarchy.py configs/optim_based/CONFIG.yaml

We start from a grid resolution of 32^3, and increase to 64^3, 128^3 and finally 256^3.

Alternatively, you can also run on a single resolution with:

python optim.py configs/optim_based/CONFIG.yaml

You might need to modify the CONFIG.yaml accordingly.

Usage for Learning-based 3D Reconstruction

Mesh Generation

To generate meshes using a trained model, use

python generate.py configs/learning_based/CONFIG.yaml

where you replace CONFIG.yaml with the correct config file.

Use a pre-trained model

The easiest way is to use a pre-trained model. You can do this by using one of the config files with postfix _pretrained.

For example, for 3D reconstruction from point clouds with outliers using our model with 7x offsets, you can simply run:

python generate.py configs/learning_based/outlier/ours_7x_pretrained.yaml

The script will automatically download the pretrained model and run the generation. You can find the outputs in the out/.../generation_pretrained folders.

Note config files are only for generation, not for training new models: when these configs are used for training, the model will be trained from scratch, but during inference our code will still use the pretrained model.

We provide the following pretrained models:

noise_small/ours.pt
noise_large/ours.pt
outlier/ours_1x.pt
outlier/ours_3x.pt
outlier/ours_5x.pt
outlier/ours_7x.pt
outlier/ours_3plane.pt

Evaluation

To evaluate a trained model, we provide the script eval_meshes.py. You can run it using:

python eval_meshes.py configs/learning_based/CONFIG.yaml

The script takes the meshes generated in the previous step and evaluates them using a standardized protocol. The output will be written to .pkl and .csv files in the corresponding generation folder that can be processed using pandas.

Training

Finally, to train a new network from scratch, simply run:

python train.py configs/learning_based/CONFIG.yaml

For available training options, please take a look at configs/default.yaml.

Implementation of ECCV20 paper: the devil is in classification: a simple framework for long-tail object detection and instance segmentation

Implementation of our ECCV 2020 paper The Devil is in Classification: A Simple Framework for Long-tail Instance Segmentation This repo contains code o

twang 98 Sep 17, 2022
Pytorch implementation for our ICCV 2021 paper "TRAR: Routing the Attention Spans in Transformers for Visual Question Answering".

TRAnsformer Routing Networks (TRAR) This is an official implementation for ICCV 2021 paper "TRAR: Routing the Attention Spans in Transformers for Visu

Ren Tianhe 49 Nov 10, 2022
DvD-TD3: Diversity via Determinants for TD3 version

DvD-TD3: Diversity via Determinants for TD3 version The implementation of paper Effective Diversity in Population Based Reinforcement Learning. Instal

3 Feb 11, 2022
Official PyTorch implementation of PICCOLO: Point-Cloud Centric Omnidirectional Localization (ICCV 2021)

Official PyTorch implementation of PICCOLO: Point-Cloud Centric Omnidirectional Localization (ICCV 2021)

16 Nov 19, 2022
Rayvens makes it possible for data scientists to access hundreds of data services within Ray with little effort.

Rayvens augments Ray with events. With Rayvens, Ray applications can subscribe to event streams, process and produce events. Rayvens leverages Apache

CodeFlare 32 Dec 25, 2022
Implementation of Learning Gradient Fields for Molecular Conformation Generation (ICML 2021).

[PDF] | [Slides] The official implementation of Learning Gradient Fields for Molecular Conformation Generation (ICML 2021 Long talk) Installation Inst

MilaGraph 117 Dec 09, 2022
A repository for storing njxzc final exam review material

文档地址,请戳我 👈 👈 👈 ☀️ 1.Reason 大三上期末复习软件工程的时候,发现其他高校在GitHub上开源了他们学校的期末试题,我很受触动。期末

GuJiakai 2 Jan 18, 2022
Python scripts for performing stereo depth estimation using the MobileStereoNet model in ONNX

ONNX-MobileStereoNet Python scripts for performing stereo depth estimation using the MobileStereoNet model in ONNX Stereo depth estimation on the cone

Ibai Gorordo 23 Nov 29, 2022
Official PyTorch implementation for paper Context Matters: Graph-based Self-supervised Representation Learning for Medical Images

Context Matters: Graph-based Self-supervised Representation Learning for Medical Images Official PyTorch implementation for paper Context Matters: Gra

49 Nov 23, 2022
The dataset of tweets pulling from Twitters with keyword: Hydroxychloroquine, location: US, Time: 2020

HCQ_Tweet_Dataset: FREE to Download. Keywords: HCQ, hydroxychloroquine, tweet, twitter, COVID-19 This dataset is associated with the paper "Understand

2 Mar 16, 2022
Rethinking of Pedestrian Attribute Recognition: A Reliable Evaluation under Zero-Shot Pedestrian Identity Setting

Pytorch Pedestrian Attribute Recognition: A strong PyTorch baseline of pedestrian attribute recognition and multi-label classification.

Jian 79 Dec 18, 2022
RoIAlign & crop_and_resize for PyTorch

RoIAlign for PyTorch This is a PyTorch version of RoIAlign. This implementation is based on crop_and_resize and supports both forward and backward on

Long Chen 530 Jan 07, 2023
A PyTorch implementation of DenseNet.

A PyTorch Implementation of DenseNet This is a PyTorch implementation of the DenseNet-BC architecture as described in the paper Densely Connected Conv

Brandon Amos 771 Dec 15, 2022
Self-Supervised Multi-Frame Monocular Scene Flow (CVPR 2021)

Self-Supervised Multi-Frame Monocular Scene Flow 3D visualization of estimated depth and scene flow (overlayed with input image) from temporally conse

Visual Inference Lab @TU Darmstadt 85 Dec 22, 2022
Probabilistic-Monocular-3D-Human-Pose-Estimation-with-Normalizing-Flows

Probabilistic-Monocular-3D-Human-Pose-Estimation-with-Normalizing-Flows This is the official implementation of the ICCV 2021 Paper "Probabilistic Mono

62 Nov 23, 2022
Single Image Deraining Using Bilateral Recurrent Network (TIP 2020)

Single Image Deraining Using Bilateral Recurrent Network Introduction Single image deraining has received considerable progress based on deep convolut

23 Aug 10, 2022
Builds a LoRa radio frequency fingerprint identification (RFFI) system based on deep learning techiniques

This project builds a LoRa radio frequency fingerprint identification (RFFI) system based on deep learning techiniques.

20 Dec 30, 2022
LSUN Dataset Documentation and Demo Code

LSUN Please check LSUN webpage for more information about the dataset. Data Release All the images in one category are stored in one lmdb database fil

Fisher Yu 426 Jan 02, 2023
https://arxiv.org/abs/2102.11005

LogME LogME: Practical Assessment of Pre-trained Models for Transfer Learning How to use Just feed the features f and labels y to the function, and yo

THUML: Machine Learning Group @ THSS 149 Dec 19, 2022
Official implementation of SIGIR'2021 paper: "Sequential Recommendation with Graph Neural Networks".

SURGE: Sequential Recommendation with Graph Neural Networks This is our TensorFlow implementation for the paper: Sequential Recommendation with Graph

FIB LAB, Tsinghua University 53 Dec 26, 2022