DIVeR: Deterministic Integration for Volume Rendering

Related tags

Deep Learningdiver
Overview

DIVeR: Deterministic Integration for Volume Rendering

This repo contains the training and evaluation code for DIVeR.

Setup

  • python 3.8
  • pytorch 1.9.0
  • pytorch-lightning 1.2.10
  • torchvision 0.2.2
  • torch-scatter 2.0.8

Dataset

Pre-trained models

Both our real-time and offline models can be found in here.

Usage

Edit configs/config.py to configure a training and setup dataset path.

To reproduce the results of the paper, replace config.py with other configuration files under the same folder.

The 'implicit' training stage takes around 40GB GPU memory and the 'implicit-explicit' stage takes around 20GB GPU memory. Decreasing the voxel grid size by a factor of 2 results in models that require around 10GB GPU memory, which causes acceptable deduction on rendering quality.

Training

To train an explicit or implicit model:

python train.py --experiment_name=EXPERIMENT_NAME \
				--device=GPU_DEVICE\
				--resume=True # if want to resume a training

After training an implicit model, the explicit model can be trained:

python train.py --experiment_name=EXPERIMENT_NAME \
				--ft=CHECKPOINT_PATH_TO_IMPLICIT_MODEL_CHECKPOINT\
				--device=GPU_DEVICE\
				--resume=True

Post processing

After the coarse model training and the fine 'implicit-explicit' model training, we perform voxel culling:

python prune.py --checkpoint_path=PATH_TO_MODEL_CHECKPOINT_FOLDER\
				--coarse_size=COARSE_IMAGE_SIZE\
				--fine_size=FINE_IMAGE_SIZE\
				--fine_ray=1 # to get rays that pass through non-empty space, 0 otherwise\
				--batch=BATCH_SIZE\
				--device=GPU_DEVICE

which stores the max-scattered 3D alpha map under model checkpoint folder as alpha_map.pt . The rays that pass through non-empty space is also stored under model checkpoint folder. For Nerf-synthetic dataset, we directly store the rays in fine_rays.npz; for Tanks&Temples and BlendedMVS, we store the mask for each pixel under folder masks which indicates the pixels (rays) to be sampled.

To convert the checkpoint file in training to pytorch model weight or serialized weight file for real-time rendering:

python convert.py --checkpoint_path=PATH_TO_MODEL_CHECKPOINT_FILE\
				  --serialize=1 # if want to build serialized weight, 0 otherwise

The converted files will be stored under the same folder as the checkpoint file, where the pytorch model weight file is named as weight.pth, and the serialized weight file is named as serialized.pth

Evaluation

To extract the offline rendered images:

python eval.py --checkpoint_path=PATH_TO_MODEL_CHECKPOINT_FILE\
			   --output_path=PATH_TO_OUTPUT_IMAGES_FOLDER\
			   --batch=BATCH_SIZE\
			   --device=GPU_DEVICE

To extract the real-time rendered images and test the mean FPS on the test sequence:

pyrhon eval_rt.py --checkpoint_path=PATH_TO_SERIALIZED_WEIGHT_FILE
				  --output_path=PATH_TO_OUPUT_IMAGES_FOLDER\
				  --decoder={32,64} # diver32, diver64\ 
				  --device=GPU_DEVICE

Resources

Citation

@misc{wu2021diver,
      title={DIVeR: Real-time and Accurate Neural Radiance Fields with Deterministic Integration for Volume Rendering}, 
      author={Liwen Wu and Jae Yong Lee and Anand Bhattad and Yuxiong Wang and David Forsyth},
      year={2021},
      eprint={2111.10427},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
Keras Image Embeddings using Contrastive Loss

Keras-Image-Embeddings-using-Contrastive-Loss Image to Embedding projection in vector space. Implementation in keras and tensorflow for custom data. B

Shravan Anand K 5 Mar 21, 2022
A big endian Gentoo port developed on a Pine64.org RockPro64

Gentoo-aarch64_be A big endian Gentoo port developed on a Pine64.org RockPro64 The endian wars are over... little endian won. As a result, it is incre

Rory Bolt 6 Dec 07, 2022
JumpDiff: Non-parametric estimator for Jump-diffusion processes for Python

jumpdiff jumpdiff is a python library with non-parametric Nadaraya─Watson estimators to extract the parameters of jump-diffusion processes. With jumpd

Rydin 28 Dec 10, 2022
Spectrum Surveying: Active Radio Map Estimation with Autonomous UAVs

Spectrum Surveying: The Python code in this repository implements the simulations and plots the figures described in the paper “Spectrum Surveying: Ac

Universitetet i Agder 2 Dec 06, 2022
.NET bindings for the Pytorch engine

TorchSharp TorchSharp is a .NET library that provides access to the library that powers PyTorch. It is a work in progress, but already provides a .NET

Matteo Interlandi 17 Aug 30, 2021
The implementation of ICASSP 2020 paper "Pixel-level self-paced learning for super-resolution"

Pixel-level Self-Paced Learning for Super-Resolution This is an official implementaion of the paper Pixel-level Self-Paced Learning for Super-Resoluti

Elon Lin 41 Dec 15, 2022
PyTorch framework for Deep Learning research and development.

Accelerated DL & RL PyTorch framework for Deep Learning research and development. It was developed with a focus on reproducibility, fast experimentati

Catalyst-Team 29 Jul 13, 2022
Laser device for neutralizing - mosquitoes, weeds and pests

Laser device for neutralizing - mosquitoes, weeds and pests (in progress) Here I will post information for creating a laser device. A warning!! How It

Ildaron 1k Jan 02, 2023
[NeurIPS-2021] Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data

MosaicKD Code for NeurIPS-21 paper "Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data" 1. Motivation Natural images share common l

ZJU-VIPA 37 Nov 10, 2022
Repository of 3D Object Detection with Pointformer (CVPR2021)

3D Object Detection with Pointformer This repository contains the code for the paper 3D Object Detection with Pointformer (CVPR 2021) [arXiv]. This wo

Zhuofan Xia 117 Jan 06, 2023
Gym for multi-agent reinforcement learning

PettingZoo is a Python library for conducting research in multi-agent reinforcement learning, akin to a multi-agent version of Gym. Our website, with

Farama Foundation 1.6k Jan 09, 2023
Minimalistic PyTorch training loop

Backbone for PyTorch training loop Will try to keep it minimalistic. pip install back from back import Bone Features Progress bar Checkpoints saving/l

Kashin 4 Jan 16, 2020
Benchmark VAE - Library for Variational Autoencoder benchmarking

Documentation pythae This library implements some of the most common (Variational) Autoencoder models. In particular it provides the possibility to pe

1.1k Jan 02, 2023
Adversarial Autoencoders

Adversarial Autoencoders (with Pytorch) Dependencies argparse time torch torchvision numpy itertools matplotlib Create Datasets python create_datasets

Felipe Ducau 188 Jan 01, 2023
Deep Learning agent of Starcraft2, similar to AlphaStar of DeepMind except size of network.

Introduction This repository is for Deep Learning agent of Starcraft2. It is very similar to AlphaStar of DeepMind except size of network. I only test

Dohyeong Kim 136 Jan 04, 2023
Using Streamlit to host a multi-page tool with model specs and classification metrics, while also accepting user input values for prediction.

Predicitng_viability Using Streamlit to host a multi-page tool with model specs and classification metrics, while also accepting user input values for

Gopalika Sharma 1 Nov 08, 2021
PyTorch implementation of "Simple and Deep Graph Convolutional Networks"

Simple and Deep Graph Convolutional Networks This repository contains a PyTorch implementation of "Simple and Deep Graph Convolutional Networks".(http

chenm 253 Dec 08, 2022
FinRL­-Meta: A Universe for Data­-Driven Financial Reinforcement Learning. 🔥

FinRL-Meta: A Universe of Market Environments. FinRL-Meta is a universe of market environments for data-driven financial reinforcement learning. Users

AI4Finance Foundation 543 Jan 08, 2023
DP-CL(Continual Learning with Differential Privacy)

DP-CL(Continual Learning with Differential Privacy) This is the official implementation of the Continual Learning with Differential Privacy. If you us

Phung Lai 3 Nov 04, 2022
SparseInst: Sparse Instance Activation for Real-Time Instance Segmentation, CVPR 2022

SparseInst 🚀 A simple framework for real-time instance segmentation, CVPR 2022 by Tianheng Cheng, Xinggang Wang†, Shaoyu Chen, Wenqiang Zhang, Qian Z

Hust Visual Learning Team 458 Jan 05, 2023