Code and models for "Pano3D: A Holistic Benchmark and a Solid Baseline for 360 Depth Estimation", OmniCV Workshop @ CVPR21.

Overview

Pano3D

A Holistic Benchmark and a Solid Baseline for 360o Depth Estimation

made-with-python Maintaner Maintaner

Streamlit Demo YouTube Video Views

Pano3D Intro

Pano3D is a new benchmark for depth estimation from spherical panoramas. We generate a dataset (using GibsonV2) and provide baselines for holistic performance assessment, offering:

  1. Primary and secondary traits metrics:
    • Direct depth performance:
      • (w)RMSE
      • (w)RMSLE
      • AbsRel
      • SqRel
      • (w)Relative accuracy (\delta) @ {1.05, 1.1, 1.25, 1.252, 1.253 }
    • Boundary discontinuity preservation:
      • Precision @ {0.25, 0.5, 1.0}m
      • Recall @ {0.25, 0.5, 1.0}m
      • Depth boundary errors of accuracy and completeness
    • Surface smoothness:
      • RMSEo
      • Relative accuracy (\alpha) @ {11.25o, 22.5o, 30o}
  2. Out-of-distribution & Zero-shot cross dataset transfer:
    • Different depth distribution test set
    • Varying scene context test set
    • Shifted camera domain test set

By disentangling generalization and assessing all depth properties, Pano3D aspires to drive progress benchmarking for 360o depth estimation.

Using Pano3D to search for a solid baseline results in an acknowledgement of exploiting complementary error terms, adding encoder-decoder skip connections and using photometric augmentations.

TODO

  • Web Demo
  • Data Download
  • Loader & Splits
  • Models Weights Download
  • Model Serve Code
  • Model Hub Code
  • Metrics Code

Demo

A publicly hosted demo of the baseline models can be found here. Using the web app, it is possible to upload a panorama and download a 3D reconstructed mesh of the scene using the derived depth map.

Note that due to the external host's caching issues, it might be necessary to refresh your browser's cache in between runs to update the 3D models.

Data

Download

To download the data, follow the instructions at vcl3d.github.io/Pano3D/download/.

Please note that getting access to the data download links is a two step process as the dataset is a derivative and compliance with the original dataset's terms and usage agreements is required. Therefore:

  1. You first need to fill in this Google Form.
  2. And, then, you need to perform an access request at each one of the Zenodo repositories (depending on which dataset partition you need):

After both these steps are completed, you will soon receive the download links for each dataset partition.

Loader

Splits

Models

Download

Inference

Serve

Metrics

Direct

Boundary

Smoothness

Results

Owner
Visual Computing Lab, Information Technologies Institute, Centre for Reseach and Technology Hellas
Computer Vision Lab in CERTH-ITI
Visual Computing Lab, Information Technologies Institute, Centre for Reseach and Technology Hellas
SmallInitEmb - LayerNorm(SmallInit(Embedding)) in a Transformer to improve convergence

SmallInitEmb LayerNorm(SmallInit(Embedding)) in a Transformer I find that when t

PENG Bo 11 Dec 25, 2022
Official repository of the paper Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision

Official repository of the paper Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision

Soubhik Sanyal 689 Dec 25, 2022
Flybirds - BDD-driven natural language automated testing framework, present by Trip Flight

Flybird | English Version 行为驱动开发(Behavior-driven development,缩写BDD),是一种软件过程的思想或者

Ctrip, Inc. 706 Dec 30, 2022
City-seeds - A random generator of cultural characteristics intended to spark ideas and help draw threads

City Seeds This is a random generator of cultural characteristics intended to sp

Aydin O'Leary 2 Mar 12, 2022
All-in-one Docker container that allows a user to explore Nautobot in a lab environment.

Nautobot Lab This container is not for production use! Nautobot Lab is an all-in-one Docker container that allows a user to quickly get an instance of

Nautobot 29 Sep 16, 2022
6D Grasping Policy for Point Clouds

GA-DDPG [website, paper] Installation git clone https://github.com/liruiw/GA-DDPG.git --recursive Setup: Ubuntu 16.04 or above, CUDA 10.0 or above, py

Lirui Wang 48 Dec 21, 2022
Malmo Collaborative AI Challenge - Team Pig Catcher

The Malmo Collaborative AI Challenge - Team Pig Catcher Approach The challenge involves 2 agents who can either cooperate or defect. The optimal polic

Kai Arulkumaran 66 Jun 29, 2022
This is the official implementation of the paper "Object Propagation via Inter-Frame Attentions for Temporally Stable Video Instance Segmentation".

[CVPRW 2021] - Object Propagation via Inter-Frame Attentions for Temporally Stable Video Instance Segmentation

Anirudh S Chakravarthy 6 May 03, 2022
Wafer Fault Detection using MlOps Integration

Wafer Fault Detection using MlOps Integration This is an end to end machine learning project with MlOps integration for predicting the quality of wafe

Sethu Sai Medamallela 0 Mar 11, 2022
PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in Power Systems

PowerGridworld provides users with a lightweight, modular, and customizable framework for creating power-systems-focused, multi-agent Gym environments that readily integrate with existing training fr

National Renewable Energy Laboratory 37 Dec 17, 2022
Learn about quantum computing and algorithm on quantum computing

quantum_computing this repo contains everything i learn about quantum computing and algorithm on quantum computing what is aquantum computing quantum

arfy slowy 8 Dec 25, 2022
Medical Insurance Cost Prediction using Machine earning

Medical-Insurance-Cost-Prediction-using-Machine-learning - Here in this project, I will use regression analysis to predict medical insurance cost for people in different regions, and based on several

1 Dec 27, 2021
Official code for "Decoupling Zero-Shot Semantic Segmentation"

Decoupling Zero-Shot Semantic Segmentation This is the official code for the arxiv. ZegFormer is the first framework that decouple the zero-shot seman

Jian Ding 108 Dec 30, 2022
(NeurIPS '21 Spotlight) IQ-Learn: Inverse Q-Learning for Imitation

Inverse Q-Learning (IQ-Learn) Official code base for IQ-Learn: Inverse soft-Q Learning for Imitation, NeurIPS '21 Spotlight IQ-Learn is an easy-to-use

Divyansh Garg 102 Dec 20, 2022
UMPNet: Universal Manipulation Policy Network for Articulated Objects

UMPNet: Universal Manipulation Policy Network for Articulated Objects Zhenjia Xu, Zhanpeng He, Shuran Song Columbia University Robotics and Automation

Columbia Artificial Intelligence and Robotics Lab 33 Dec 03, 2022
TinyML Cookbook, published by Packt

TinyML Cookbook This is the code repository for TinyML Cookbook, published by Packt. Author: Gian Marco Iodice Publisher: Packt About the book This bo

Packt 93 Dec 29, 2022
Deep Learning to Improve Breast Cancer Detection on Screening Mammography

Shield: This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. Deep Learning to Improve Breast

Li Shen 305 Jan 03, 2023
PipeTransformer: Automated Elastic Pipelining for Distributed Training of Large-scale Models

PipeTransformer: Automated Elastic Pipelining for Distributed Training of Large-scale Models This repository is the official implementation of the fol

DistributedML 41 Dec 06, 2022
Fully Convolutional DenseNets for semantic segmentation.

Introduction This repo contains the code to train and evaluate FC-DenseNets as described in The One Hundred Layers Tiramisu: Fully Convolutional Dense

485 Nov 26, 2022