Multi View Stereo on Internet Images

Overview

Evaluating MVS in a CPC Scenario

This repository contains the set of artficats used for the ENGN8601/8602 research project. The thesis emphasizes on the following aspects:

  • Evaluating and Analysing the performance of existing learning-based MVS networks on internet images or in a CPC scenario.
  • Proposing a novel mask estimation module and depth estimation (with depth alignment) framework to estimate depth values of foreground objects.
  • Fusing the depthmaps estimated by the proposed methodology to compute complete point clouds (including foreground objects)

Installation

Recommended: python 3.7.7, cudatoolkit 10.2.* and conda.

The python libraries required are provided in the requirements.txt file. You can install the environment and necessary modules as follows or use your own approach:

Create a new conda environment and activate it:

conda create -n mvs
conda activate mvs

Install requirements.txt and opencv & pytorch separately (make sure pip is installed):

pip install -r requirements.txt
pip install opencv-python
pip install torch==1.8.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html

Data

Download the validation dataset extracted from MegaDepth dataset here and extract the all the images from the dataset to ref_images folder as subdirectories (create ref_images if it doesn't exist).

Usage

All the results have been included in the downloaded dataset already for ease of access to the results. The directory structure is given and explained as follows. The image subdirectories include reference images, npy files containing camera information, entropies, depthmaps (monocular, estimated and ground truth) etc.

  • All the images inside the ref_images directory contain a grid_outputs subdirectory which contain the best masks estimated for the reference image. This subdirectory also contains the plots with the .npy files for visualization.
  • All the images inside the ref_images directory contain ply and ply_gt subdirectories, which contain the 3D world points and .ply files depicting the estimated point cloud of that scene reconstructed by the proposed method.
  • The final_fused_scenes folder contains the point cloud generated by fusing multiple depthmaps obtained from different views of the same scene.

NOTE: You do not need to run the following steps in a sequence since required intermediate results are already provided in the images (such as entropy maps etc.), you can run any step directly.

1. Mask Estimation

Open the terminal and run the following command:

python masking.py

The 10 best masks with lowest binary cross entropy loss and IoU for each reference image is computed and saved inside a grid_outputs subdirectory inside each image directory. You can view the mask visualizations which are saved as .png. The masks are also saved as .npy files.

2. Calculating Error Metrics

To calculate the EPE, 1px and 3px errors between the estimated depthmaps and ground truth depthmaps, run the following command:

python calc_errors.py

3. 3D Reconstruction - Individual Point Clouds

To reconstruct ground truth point clouds and the estimated point clouds with foreground objects for each individual reference image, run the following command:

python pfm2ply_aligned.py                   # Point Clouds from Estimated Depthmaps (with foreground)
python pfm2ply_aligned_gt.py                # Point Clouds from Ground Truth Depthmaps

The point clouds will be saved in the ply and ply_gt image subdirectories respectively as .ply files along with the vertices of these point clouds saved as vertices.npy. This also generates the aligned absolute depthmap and saves the visualization along with the monocular depthmap estimated via the monocular depth estimation network inside the image directories.

4. Generating Scene Point Clouds

Since step 3 comptues individual point clouds, the next task is to merge the vertices of each individual point cloud to generate the point cloud for an entire scene. Run the following command:

python merge2ply.py

You can specify the set of images to be used for reconstructing each scene by editing the merge2ply file. All the scene point clouds are saved in final_fused_scenes folder.

Visualizations of Outputs

1. Mask Estimation

masks

2. 3D Reconstruction of Invidiual Depthmaps

3dr

3. Merged Point Clouds

3dr scenes

Supporting Repositories

I would like to give credit to the following repositories for assisting me in computing intermediate results necessary for the thesis:

Thank you!

Owner
Namas Bhandari
Machine Learning/Deep Learning/AI Enthusiast
Namas Bhandari
Este software fornece interface gráfica para o escputil e tem por finalidade testar e fazer limpeza no cabeçote de impressão....

PrinterTools O que é PrinterTools? PrinterTools é uma ferramenta gráfica que usa o escputil para testar e fazer limpeza de cabeçote de impressão em si

Elizeu Barbosa Abreu 1 Dec 21, 2021
An OpenSource crowd-sourced cooking recipes website

An OpenSource crowd-sourced cooking recipes website

21 Jul 31, 2022
Convert Beat Saber maps to Tesla light shows!

Tesla x Beat Saber - Light Show Converter Convert Beat Saber maps to Tesla light shows! This project requires FFMPEG and all packages from requirement

HLVM 20 Dec 21, 2022
Dotfiles for my configurations!

Dotfiles Repo Welcome! Over here, you can find my dotfiles for various applications, including QTile, Alacritty, Kitty, LunarVim, and more! Make sure

class PythonAddict 3 Jan 10, 2022
AIO solution for SSIS students

ssis.bit AIO solution for SSIS students Hardware CircuitPython supports more than 200 different boards. Locally available is the TTGO T8 ESP32-S2 ST77

3 Jun 05, 2022
An implementation of multimap with per-item expiration backed up by Redis.

MultiMapWithTTL An implementation of multimap with per-item expiration backed up by Redis. Documentation: https://loggi.github.io/python-multimapwitht

Loggi 2 Jan 17, 2022
A python library what works with numbers.

pynum A python library what works with numbers. Prime Prime class have everithing you want about prime numbers. check_prime The check_prime method is

Mohammad Mahdi Paydar Puya 1 Jan 07, 2022
python3 scrip for case conversion of source code files writen in fixed form fortran

convert_FORTRAN_case python3 scrip for case conversion of source code files writen in fixed form fortran python3 scrip for case conversion of source c

7 Sep 20, 2022
Calibre Libgen Non-fiction / Sci-tech store plugin

CalibreLibgenSci A Libgen Non-Fiction/Sci-tech store plugin for Calibre Installation Download the latest zip file release from here Open Calibre Navig

IDDQD 9 Dec 27, 2022
program to store and update pokemons using SQL and Flask

Pokemon SQL and Flask Pokemons api in python. Technologies flask pymysql Description PokeCorp is a company that tracks pokemon and their trainers arou

Sara Hindy Salfer 1 Oct 20, 2021
[x]it! support for working with todo and check list files in Sublime Text

[x]it! for Sublime Text This Sublime Package provides syntax-highlighting, shortcuts, and auto-completions for [x]it! files. Features Syntax highlight

Jan Heuermann 18 Sep 19, 2022
Tutorials for on-ramping to StarkNet

Full-Stack StarkNet Repo containing the code for a short tutorial series I wrote while diving into StarkNet and learning Cairo. Aims to onramp existin

Sam Barnes 71 Dec 07, 2022
Experiments with Tox plugin system

The project is an attempt to add to the tox some missing out of the box functionality. Basically it is just an extension for the tool that will be loa

Volodymyr Vitvitskyi 30 Nov 26, 2022
An example project that shows how to check if a certain macro is active in a file.

PlatformIO Check Compiler Flags Example Description Demonstrates the usage of an extra script and a special compilter invocation to get the active mac

Maximilian Gerhardt 1 Oct 28, 2021
Automatically find solutions when your Python code encounters an issue.

What The Python?! Helping you find answers to the errors Python spits out. Installation You can find the source code on GitHub at: https://github.com/

What The Python?! 139 Dec 14, 2022
Weakly-Divisable - Takes an interger and seee if it is weakly divisible by seven

Weakly Divisble Project by Diana Arce-Hernandez, Ryan McAlpine, and Rommel Ravan

Diana Arce-Hernandez 1 Jan 12, 2022
Online HackerRank problem solving challenges

LinkedListHackerRank Online HackerRank problem solving challenges This challenge is part of a tutorial track by MyCodeSchool You are given the pointer

Sefineh Tesfa 1 Nov 21, 2021
A Python script to delete movies with a certain tag after a certain amount of days.

radarr_autodelete Simple script, which deletes movies with a specific tag after a certain amount of days Pip Packages pip3 install pyarr python-dotenv

7 Dec 06, 2022
Fixed waypoint(pose) navigation for turtlebot simulation.

Turtlebot-NavigationStack-Fixed-Waypoints fixed waypoint(pose) navigation for turtlebot simulation. Task Details Task Permformed using Navigation Stac

Shanmukha Vishnu 1 Apr 08, 2022
Odoo. Open Source Apps To Grow Your Business.

Odoo Odoo is a suite of web based open source business apps. The main Odoo Apps include an Open Source CRM, Website Builder, eCommerce, Warehouse Mana

Odoo 27.6k Jan 09, 2023