We evaluate our method on different datasets (including ShapeNet, CUB-200-2011, and Pascal3D+) and achieve state-of-the-art results, outperforming all the other supervised and unsupervised methods and 3D representations, all in terms of performance, accuracy, and training time.

Last update: Dec 27, 2022

Overview

An Effective Loss Function for Generating 3D Models from Single 2D Image without Rendering

Papers with code | Paper

Nikola Zubić Pietro Lio

University of Novi Sad University of Cambridge

AIAI 2021

Citation

Besides AIAI 2021, our paper is in a Springer's book entitled "Artificial Intelligence Applications and Innovations": link

Please, cite our paper if you find this code useful for your research.

@article{zubic2021effective,
  title={An Effective Loss Function for Generating 3D Models from Single 2D Image without Rendering},
  author={Zubi{\'c}, Nikola and Li{\`o}, Pietro},
  journal={arXiv preprint arXiv:2103.03390},
  year={2021}
}

Prerequisites

Download code:
Git clone the code with the following command:

git clone https://github.com/NikolaZubic/2dimageto3dmodel.git

Open the project with Conda Environment (Python 3.7)

Install packages:

conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch

Then git clone Kaolin library in the root (2dimageto3dmodel) folder with the following commit and run the following commands:

cd kaolin
python setup.py install
pip install --no-dependencies nuscenes-devkit opencv-python-headless scikit-learn joblib pyquaternion cachetools
pip install packaging

Run the program

Run the following commands from the root/code/ (2dimageto3dmodel/code/) directory:

python main.py --dataset cub --batch_size 16 --weights pretrained_weights_cub --save_results

for the CUB Birds Dataset.

python main.py --dataset p3d --batch_size 16 --weights pretrained_weights_p3d --save_results

for the Pascal 3D+ Dataset.

The results will be saved at 2dimageto3dmodel/code/results/ path.

Continue training

To continue the training process:
Run the following commands (without --save_results) from the root/code/ (2dimageto3dmodel/code/) directory:

python main.py --dataset cub --batch_size 16 --weights pretrained_weights_cub

for the CUB Birds Dataset.

python main.py --dataset p3d --batch_size 16 --weights pretrained_weights_p3d

for the Pascal 3D+ Dataset.

License

MIT

Acknowledgment

This idea has been built based on the architecture of Insafutdinov & Dosovitskiy.
Poisson Surface Reconstruction was used for Point Cloud to 3D Mesh transformation.
The GAN architecture (used for texture mapping) is a mixture of Xian's TextureGAN and Li's GAN.

Comments

Where is cmr_data?

Keep running into this issue from cmr_data.p3d import P3dDataset and from cmr_data.p3d import CUBDataset

but you do not have these files in your repo. I tried using cub_200_2011_dataset.py but it does not take in the same number of arguments as the CUBDataset class used in run_reconstruction.py.

opened by achhabria7 6
ModuleNotFoundError: No module named 'kaolin.graphics'

Pascal 3D+ dataset with 4722 images is successfully loaded.

Traceback (most recent call last): File "main.py", line 149, in <module> from rendering.renderer import Renderer File "/home/ujjawal/my_work/object_recon/2d3d/code/rendering/renderer.py", line 1, in <module> from kaolin.graphics.dib_renderer.rasterizer import linear_rasterizer ModuleNotFoundError: No module named kaolin.graphics

I also downloaded the graphics folder from here https://github.com/NVIDIAGameWorks/kaolin/tree/e7e513173bd4159ae45be6b3e156a3ad156a3eb9 and tried to place in the graphics folder in the kaolin folder locally and here is the error Traceback (most recent call last): File "main.py", line 149, in <module> from rendering.renderer import Renderer File "/home/ujjawal/my_work/object_recon/2d3d/code/rendering/renderer.py", line 1, in <module> from kaolin.graphics.dib_renderer.rasterizer import linear_rasterizer File "/usr/local/lib/python3.6/dist-packages/kaolin-0.9.0-py3.6-linux-x86_64.egg/kaolin/graphics/__init__.py", line 2, in <module> File "/usr/local/lib/python3.6/dist-packages/kaolin-0.9.0-py3.6-linux-x86_64.egg/kaolin/graphics/nmr/__init__.py", line 1, in <module> File "/usr/local/lib/python3.6/dist-packages/kaolin-0.9.0-py3.6-linux-x86_64.egg/kaolin/graphics/nmr/rasterizer.py", line 30, in <module> ImportError: cannot import name rasterize_cuda

opened by ujjawalcse 6
No module named 'models.reconstruction'

Dear NikolaZubic :
Thanks for you updated the code recently. Did you put the reconstruction.py in the models folder？When I run “python run_reconstruction.py --name pretrained_reconstruction_cub --dataset cub --batch_size 10 --generate_pseudogt” it display
No module named 'models.reconstruction.

opened by lw0210 2
inference with single RGB pictures

Hi, I am interested with your work, it is wonderful, and I want to use my own picture to test the model, could you provided the pretrained model and inference scripts.

opened by 523997931 2
$can't find the pseudogt_512*512\.npz file$

can't find the pseudogt_512*512\.npz file

Dear NikolaZubic: I want to quote your paper, but I can't find the pseudogt_512512.npz file and can't reproduce it. Can you give me the pseudogt_512512.npz file and help me reproduce it? Thanks

opened by Yangfuha 1
ValueError: Training a model requires the pseudo-ground-truth to be setup beforehand.

I recently read your paper and was very interested in it . I want to reproduce the code of this paper. When I followed your instructions, I found it difficult for me to run the commands（python main.py --dataset cub --batch_size 16 --weights pretrained_weights_cub and python main.py --dataset p3d --batch_size 16 --weights pretrained_weights_p3d.）.And the program displayed a value error that training a model requires the pseudo-ground-truth to be setup beforehand. And I don’t know how to solve the problem, so I turn to you for help.I'm sorry to bother you, but I'really eager to solve the problem. I hope to get your reply.Thank you!

opened by lw0210 1
Added step: switch to the correct correct Kaolin branch

This step will help others to avoid the "ModuleNotFoundError: No module named kaolin.graphics" error.

Fix to issue: https://github.com/NikolaZubic/2dimageto3dmodel/issues/2

opened by ricklentz 1
Shapenet V2 not training

Great work guys. I was able to run the code on CUB dataset. But when I tried to run training_test_shape_net.py on Shape Net v2 chair class I'm getting errors because of missing files, unmatched file names, etc.

So it would be helpful if you provide Shapenet Dataset Folder structure and files(images, masks) description or a sample folder and clear instructions for training the model shapenet dataset. And also if possible give pre-trained weights for the Shape net dataset models

Thank you

opened by girishdhegde 0
Pretrained model

Hi, I find it hard to understand how to train the model on ShapeNet. It would be very helpful if you can provide a pretrained model on ShapeNet planes (I need it to test the performance in my project). If the pretrained models are not available, it would also be helpful to introduce me of how to train the model on ShapeNet.

opened by YYYYYHC 0
How can I train on the boat set of the Pascal 3D+ dataset

I find the data of trainning such as "python run_reconstruction.py --name pretrained_reconstruction_p3d --dataset p3d --optimize_z0 --batch_size 50 --tensorboard" using the data of car.mat in sfm and data folder.Even if I rename the .mat to boat.mat and using the boat imageNet in Pascal 3D+ dataset,I find the shape of the result is more like a car not a boat.So I am wondering how to train the boat set.

opened by lisentao 0
Custom Dataset

Hi!

Love the work you guys have done. I am currently conducting a research. Could you please tell me how I would train on a custom dataset and how I would infer an image or create a 3d model out an image with pretrained weights that you have provided?

opened by mahnoor-fatima-saad 0
How do I make my own dataset？

Dear NikolaZubic： I want to use my own data set to replace the cub or P3D data set for training. Do you have any attention or requirements for images when making data sets?

opened by lw0210 0

Releases(metadata)

metadata(Oct 30, 2021)

Many users have complained about problems while loading and training on specific sets.

With this release, we provide you with full metadata for this problem that includes all the pre-trained models, along with the cache directory (poses and precomputed statistics).

Zip files must be extracted to the root directory of the cloned repository.
Source code(tar.gz)
Source code(zip)
cache_datasets.zip(61.19 MB)
pretrained_models.zip(604.57 MB)

Owner

Nikola Zubić

Interested in Artificial intelligence, Visual Computing and Cognitive science. For future AI projects: @reinai

GitHub Repository

This repository is the official implementation of Using Time-Series Privileged Information for Provably Efficient Learning of Prediction Models

Using Time-Series Privileged Information for Provably Efficient Learning of Prediction Models Link to paper Abstract We study prediction of future out

2 Aug 19, 2022

Deep Learning applied to Integral data analysis

DeepIntegralCompton Deep Learning applied to Integral data analysis Module installation Move to the root directory of the project and execute : pip in

1 Dec 10, 2021

Implementation of Fast Transformer in Pytorch

Fast Transformer - Pytorch Implementation of Fast Transformer in Pytorch. This only work as an encoder. Yannic video AI Epiphany Install $ pip install

167 Dec 27, 2022

Company clustering with K-means/GMM and visualization with PCA, t-SNE, using SSAN relation extraction

RE results graph visualization and company clustering Installation pip install -r requirements.txt python -m nltk.downloader stopwords python3.7 main.

1 Oct 06, 2022

A bunch of random PyTorch models using PyTorch's C++ frontend

PyTorch Deep Learning Models using the C++ frontend Gettting started Clone the repo 1. https://github.com/mrdvince/pytorchcpp 2. cd fashionmnist or

0 Jul 13, 2021

Anomaly detection in multi-agent trajectories: Code for training, evaluation and the OpenAI highway simulation.

Anomaly Detection in Multi-Agent Trajectories for Automated Driving This is the official project page including the paper, code, simulation, baseline

12 Dec 02, 2022

Aerial Imagery dataset for fire detection: classification and segmentation (Unmanned Aerial Vehicle (UAV))

Aerial Imagery dataset for fire detection: classification and segmentation using Unmanned Aerial Vehicle (UAV) Title FLAME (Fire Luminosity Airborne-b

79 Jan 06, 2023

Yggdrasil - A simplistic bot designed to streamline your server experience

Ygggdrasil A simplistic bot designed to streamline your server experience. Desig

1 Dec 14, 2022

RM Operation can equivalently convert ResNet to VGG, which is better for pruning; and can help RepVGG perform better when the depth is large.

RMNet: Equivalently Removing Residual Connection from Networks This repository is the official implementation of "RMNet: Equivalently Removing Residua

184 Jan 04, 2023

We evaluate our method on different datasets (including ShapeNet, CUB-200-2011, and Pascal3D+) and achieve state-of-the-art results, outperforming all the other supervised and unsupervised methods and 3D representations, all in terms of performance, accuracy, and training time.

Related tags

Overview

An Effective Loss Function for Generating 3D Models from Single 2D Image without Rendering

Papers with code | Paper

Citation

Prerequisites

Run the program

Continue training

License

Acknowledgment

Comments

Releases(metadata)

metadata(Oct 30, 2021)

Owner

Nikola Zubić

This repository is the official implementation of Using Time-Series Privileged Information for Provably Efficient Learning of Prediction Models

Deep Learning applied to Integral data analysis

Implementation of Fast Transformer in Pytorch

Company clustering with K-means/GMM and visualization with PCA, t-SNE, using SSAN relation extraction

A bunch of random PyTorch models using PyTorch's C++ frontend

Anomaly detection in multi-agent trajectories: Code for training, evaluation and the OpenAI highway simulation.

Aerial Imagery dataset for fire detection: classification and segmentation (Unmanned Aerial Vehicle (UAV))

Yggdrasil - A simplistic bot designed to streamline your server experience

RM Operation can equivalently convert ResNet to VGG, which is better for pruning; and can help RepVGG perform better when the depth is large.

Rethinking the U-Net architecture for multimodal biomedical image segmentation

(Personalized) Page-Rank computation using PyTorch

Demo code for paper "Learning optical flow from still images", CVPR 2021.

[ICLR'21] Counterfactual Generative Networks

Moving Object Segmentation in 3D LiDAR Data: A Learning-based Approach Exploiting Sequential Data

Reimplementation of Learning Mesh-based Simulation With Graph Networks

Learning Representations that Support Robust Transfer of Predictors

A PyTorch library and evaluation platform for end-to-end compression research

Generalized and Efficient Blackbox Optimization System.

Deep Learning for Time Series Classification

CNNs for Sentence Classification in PyTorch