DIVeR: Deterministic Integration for Volume Rendering

Related tags

Deep Learningdiver
Overview

DIVeR: Deterministic Integration for Volume Rendering

This repo contains the training and evaluation code for DIVeR.

Setup

  • python 3.8
  • pytorch 1.9.0
  • pytorch-lightning 1.2.10
  • torchvision 0.2.2
  • torch-scatter 2.0.8

Dataset

Pre-trained models

Both our real-time and offline models can be found in here.

Usage

Edit configs/config.py to configure a training and setup dataset path.

To reproduce the results of the paper, replace config.py with other configuration files under the same folder.

The 'implicit' training stage takes around 40GB GPU memory and the 'implicit-explicit' stage takes around 20GB GPU memory. Decreasing the voxel grid size by a factor of 2 results in models that require around 10GB GPU memory, which causes acceptable deduction on rendering quality.

Training

To train an explicit or implicit model:

python train.py --experiment_name=EXPERIMENT_NAME \
				--device=GPU_DEVICE\
				--resume=True # if want to resume a training

After training an implicit model, the explicit model can be trained:

python train.py --experiment_name=EXPERIMENT_NAME \
				--ft=CHECKPOINT_PATH_TO_IMPLICIT_MODEL_CHECKPOINT\
				--device=GPU_DEVICE\
				--resume=True

Post processing

After the coarse model training and the fine 'implicit-explicit' model training, we perform voxel culling:

python prune.py --checkpoint_path=PATH_TO_MODEL_CHECKPOINT_FOLDER\
				--coarse_size=COARSE_IMAGE_SIZE\
				--fine_size=FINE_IMAGE_SIZE\
				--fine_ray=1 # to get rays that pass through non-empty space, 0 otherwise\
				--batch=BATCH_SIZE\
				--device=GPU_DEVICE

which stores the max-scattered 3D alpha map under model checkpoint folder as alpha_map.pt . The rays that pass through non-empty space is also stored under model checkpoint folder. For Nerf-synthetic dataset, we directly store the rays in fine_rays.npz; for Tanks&Temples and BlendedMVS, we store the mask for each pixel under folder masks which indicates the pixels (rays) to be sampled.

To convert the checkpoint file in training to pytorch model weight or serialized weight file for real-time rendering:

python convert.py --checkpoint_path=PATH_TO_MODEL_CHECKPOINT_FILE\
				  --serialize=1 # if want to build serialized weight, 0 otherwise

The converted files will be stored under the same folder as the checkpoint file, where the pytorch model weight file is named as weight.pth, and the serialized weight file is named as serialized.pth

Evaluation

To extract the offline rendered images:

python eval.py --checkpoint_path=PATH_TO_MODEL_CHECKPOINT_FILE\
			   --output_path=PATH_TO_OUTPUT_IMAGES_FOLDER\
			   --batch=BATCH_SIZE\
			   --device=GPU_DEVICE

To extract the real-time rendered images and test the mean FPS on the test sequence:

pyrhon eval_rt.py --checkpoint_path=PATH_TO_SERIALIZED_WEIGHT_FILE
				  --output_path=PATH_TO_OUPUT_IMAGES_FOLDER\
				  --decoder={32,64} # diver32, diver64\ 
				  --device=GPU_DEVICE

Resources

Citation

@misc{wu2021diver,
      title={DIVeR: Real-time and Accurate Neural Radiance Fields with Deterministic Integration for Volume Rendering}, 
      author={Liwen Wu and Jae Yong Lee and Anand Bhattad and Yuxiong Wang and David Forsyth},
      year={2021},
      eprint={2111.10427},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
Discord Multi Tool that focuses on design and easy usage

Multi-Tool-v1.0 Discord Multi Tool that focuses on design and easy usage Delete webhook Block all friends Spam webhook Modify webhook Webhook info Tok

Lodi#0001 24 May 23, 2022
Public repo for the ICCV2021-CVAMD paper "Is it Time to Replace CNNs with Transformers for Medical Images?"

Is it Time to Replace CNNs with Transformers for Medical Images? Accepted at ICCV-2021: Workshop on Computer Vision for Automated Medical Diagnosis (C

Christos Matsoukas 80 Dec 27, 2022
Multiband spectro-radiometric satellite image analysis with K-means cluster algorithm

Multi-band Spectro Radiomertric Image Analysis with K-means Cluster Algorithm Overview Multi-band Spectro Radiomertric images are images comprising of

Chibueze Henry 6 Mar 16, 2022
a reimplementation of Holistically-Nested Edge Detection in PyTorch

pytorch-hed This is a personal reimplementation of Holistically-Nested Edge Detection [1] using PyTorch. Should you be making use of this work, please

Simon Niklaus 375 Dec 06, 2022
This is the repository for our paper Ditch the Gold Standard: Re-evaluating Conversational Question Answering

Ditch the Gold Standard: Re-evaluating Conversational Question Answering This is the repository for our paper Ditch the Gold Standard: Re-evaluating C

Princeton Natural Language Processing 38 Dec 16, 2022
Training neural models with structured signals.

Neural Structured Learning in TensorFlow Neural Structured Learning (NSL) is a new learning paradigm to train neural networks by leveraging structured

955 Jan 02, 2023
Source Code for ICSE 2022 Paper - ``Can We Achieve Fairness Using Semi-Supervised Learning?''

Fair-SSL Source Code for ICSE 2022 Paper - Can We Achieve Fairness Using Semi-Supervised Learning? Ethical bias in machine learning models has become

1 Dec 18, 2021
An AFL implementation with UnTracer (our coverage-guided tracer)

UnTracer-AFL This repository contains an implementation of our prototype coverage-guided tracing framework UnTracer in the popular coverage-guided fuz

113 Dec 17, 2022
Joint-task Self-supervised Learning for Temporal Correspondence (NeurIPS 2019)

Joint-task Self-supervised Learning for Temporal Correspondence Project | Paper Overview Joint-task Self-supervised Learning for Temporal Corresponden

Sifei Liu 167 Dec 14, 2022
Deep Learning to Create StepMania SM FIles

StepCOVNet Running Audio to SM File Generator Currently only produces .txt files. Use SMDataTools to convert .txt to .sm python stepmania_note_generat

Chimezie Iwuanyanwu 8 Jan 08, 2023
Self-training with Weak Supervision (NAACL 2021)

This repo holds the code for our weak supervision framework, ASTRA, described in our NAACL 2021 paper: "Self-Training with Weak Supervision"

Microsoft 148 Nov 20, 2022
Official PyTorch implementation of RIO

Image-Level or Object-Level? A Tale of Two Resampling Strategies for Long-Tailed Detection Figure 1: Our proposed Resampling at image-level and obect-

NVIDIA Research Projects 17 May 20, 2022
MISSFormer: An Effective Medical Image Segmentation Transformer

MISSFormer Code for paper "MISSFormer: An Effective Medical Image Segmentation Transformer". Please read our preprint at the following link: paper_add

Fong 22 Dec 24, 2022
Code for paper "A Critical Assessment of State-of-the-Art in Entity Alignment" (https://arxiv.org/abs/2010.16314)

A Critical Assessment of State-of-the-Art in Entity Alignment This repository contains the source code for the paper A Critical Assessment of State-of

Max Berrendorf 16 Oct 14, 2022
The MLOps platform for innovators 🚀

​ DS2.ai is an integrated AI operation solution that supports all stages from custom AI development to deployment. It is an AI-specialized platform service that collects data, builds a training datas

9 Jan 03, 2023
This repository contains an overview of important follow-up works based on the original Vision Transformer (ViT) by Google.

This repository contains an overview of important follow-up works based on the original Vision Transformer (ViT) by Google.

75 Dec 02, 2022
Streaming Anomaly Detection Framework in Python (Outlier Detection for Streaming Data)

Python Streaming Anomaly Detection (PySAD) PySAD is an open-source python framework for anomaly detection on streaming multivariate data. Documentatio

Selim Firat Yilmaz 181 Dec 18, 2022
Code for "Unsupervised Source Separation via Bayesian inference in the latent domain"

LQVAE-separation Code for "Unsupervised Source Separation via Bayesian inference in the latent domain" Paper Samples GT Compressed Separated Drums GT

Michele Mancusi 30 Oct 25, 2022
Facial expression detector

A tensorflow convolutional neural network model to detect facial expressions.

Carlos Tardón Rubio 5 Apr 20, 2022
RLDS stands for Reinforcement Learning Datasets

RLDS RLDS stands for Reinforcement Learning Datasets and it is an ecosystem of tools to store, retrieve and manipulate episodic data in the context of

Google Research 135 Jan 01, 2023