Spherical CNNs

Related tags

Deep Learnings2cnn
Overview

Spherical CNNs

Equivariant CNNs for the sphere and SO(3) implemented in PyTorch

Equivariance

Overview

This library contains a PyTorch implementation of the rotation equivariant CNNs for spherical signals (e.g. omnidirectional images, signals on the globe) as presented in [1]. Equivariant networks for the plane are available here.

Dependencies

(commands to install all the dependencies on a new conda environment)

conda create --name cuda9 python=3.6 
conda activate cuda9

# s2cnn deps
#conda install pytorch torchvision cuda90 -c pytorch # get correct command line at http://pytorch.org/
conda install -c anaconda cupy  
pip install pynvrtc joblib

# lie_learn deps
conda install -c anaconda cython  
conda install -c anaconda requests  

# shrec17 example dep
conda install -c anaconda scipy  
conda install -c conda-forge rtree shapely  
conda install -c conda-forge pyembree  
pip install "trimesh[easy]"  

Installation

To install, run

$ python setup.py install

Usage

Please have a look at the examples.

Please cite [1] in your work when using this library in your experiments.

Design choices for Spherical CNN Architectures

Spherical CNNs come with different choices of grids and grid hyperparameters which are on the first look not obviously related to those of conventional CNNs. The s2_near_identity_grid and so3_near_identity_grid are the preferred choices since they correspond to spatially localized kernels, defined at the north pole and rotated over the sphere via the action of SO(3). In contrast, s2_equatorial_grid and so3_equatorial_grid define line-like (or ring-like) kernels around the equator.

To clarify the possible parameter choices for s2_near_identity_grid:

max_beta:

Adapts the size of the kernel as angle measured from the north pole. Conventional CNNs on flat space usually use a fixed kernel size but pool the signal spatially. This spatial pooling gives the kernels in later layers an effectively increased field of view. One can emulate a pooling by a factor of 2 in spherical CNNs by decreasing the signal bandwidth by 2 and increasing max_beta by 2.

n_beta:

Number of rings of the kernel around the equator, equally spaced in [β=0, β=max_beta]. The choice n_beta=1 corresponds to a small 3x3 kernel in conv2d since in both cases the resulting kernel consists of one central pixel and one ring around the center.

n_alpha:

Gives the number of learned parameters of the rings around the pole. These values are per default equally spaced on the azimuth. A sensible number of values depends on the bandwidth and max_beta since a higher resolution or spatial extent allow to sample more fine kernels without producing aliased results. In practice this value is typically set to a constant, low value like 6 or 8. A reduced bandwidth of the signal is thereby counteracted by an increased max_beta to emulate spatial pooling.

The so3_near_identity_grid has two additional parameters max_gamma and n_gamma. SO(3) can be seen as a (principal) fiber bundle SO(3)→S² with the sphere S² as base space and fiber SO(2) attached to each point. The additional parameters control the grid on the fiber in the following way:

max_gamma:

The kernel spans over the fiber SO(2) between γ∈[0, max_gamma]. The fiber SO(2) encodes the kernel responses for every sampled orientation at a given position on the sphere. Setting max_gamma≨2π results in the kernel not seeing the responses of all kernel orientations simultaneously and is in general unfavored. Steerable CNNs [3] usually always use max_gamma=2π.

n_gamma:

Number of learned parameters on the fiber. Typically set equal to n_alpha, i.e. to a low value like 6 or 8.

See the deep model of the MNIST example for an example of how to adapt these parameters over layers.

Feedback

For questions and comments, feel free to contact us: geiger.mario (gmail), taco.cohen (gmail), jonas (argmin.xyz).

License

MIT

References

[1] Taco S. Cohen, Mario Geiger, Jonas Köhler, Max Welling, Spherical CNNs. International Conference on Learning Representations (ICLR), 2018.

[2] Taco S. Cohen, Mario Geiger, Jonas Köhler, Max Welling, Convolutional Networks for Spherical Signals. ICML Workshop on Principled Approaches to Deep Learning, 2017.

[3] Taco S. Cohen, Mario Geiger, Maurice Weiler, Intertwiners between Induced Representations (with applications to the theory of equivariant neural networks), ArXiv preprint 1803.10743, 2018.

Owner
Jonas Köhler
Jonas Köhler
Deep Learning for Natural Language Processing SS 2021 (TU Darmstadt)

Deep Learning for Natural Language Processing SS 2021 (TU Darmstadt) Task Training huge unsupervised deep neural networks yields to strong progress in

2 Aug 05, 2022
Official PyTorch implementation of UACANet: Uncertainty Aware Context Attention for Polyp Segmentation

UACANet: Uncertainty Aware Context Attention for Polyp Segmentation Official pytorch implementation of UACANet: Uncertainty Aware Context Attention fo

Taehun Kim 85 Dec 14, 2022
Anime Face Detector using mmdet and mmpose

Anime Face Detector This is an anime face detector using mmdetection and mmpose. (To avoid copyright issues, I use generated images by the TADNE model

198 Jan 07, 2023
PyG (PyTorch Geometric) - A library built upon PyTorch to easily write and train Graph Neural Networks (GNNs)

PyG (PyTorch Geometric) is a library built upon PyTorch to easily write and train Graph Neural Networks (GNNs) for a wide range of applications related to structured data.

PyG 16.5k Jan 08, 2023
CSKG is a commonsense knowledge graph that combines seven popular sources into a consolidated representation

CSKG: The CommonSense Knowledge Graph CSKG is a commonsense knowledge graph that combines seven popular sources into a consolidated representation: AT

USC ISI I2 85 Dec 12, 2022
A novel Engagement Detection with Multi-Task Training (ED-MTT) system

A novel Engagement Detection with Multi-Task Training (ED-MTT) system which minimizes MSE and triplet loss together to determine the engagement level of students in an e-learning environment.

Onur Çopur 12 Nov 11, 2022
SemEval2022 Patronizing and Condescending Language (PCL) Detection

SemEval2022 Patronizing and Condescending Language (PCL) Detection This task is from SemEval 2022. What is Patronizing and Condescending Language (PCL

Daniel Saeedi 0 Aug 05, 2022
[NeurIPS'21] "AugMax: Adversarial Composition of Random Augmentations for Robust Training" by Haotao Wang, Chaowei Xiao, Jean Kossaifi, Zhiding Yu, Animashree Anandkumar, and Zhangyang Wang.

AugMax: Adversarial Composition of Random Augmentations for Robust Training Haotao Wang, Chaowei Xiao, Jean Kossaifi, Zhiding Yu, Anima Anandkumar, an

VITA 112 Nov 07, 2022
22 Oct 14, 2022
Toolchain to build Yoshi's Island from source code

Project-Y Toolchain to build Yoshi's Island (J) V1.0 from source code, by MrL314 Last updated: September 17, 2021 Setup To begin, download this toolch

MrL314 19 Apr 18, 2022
LightLog is an open source deep learning based lightweight log analysis tool for log anomaly detection.

LightLog Introduction LightLog is an open source deep learning based lightweight log analysis tool for log anomaly detection. Function description [BG

25 Dec 17, 2022
Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Zhengzhong Tu 5 Sep 16, 2022
PyTorch implementation of "PatchGame: Learning to Signal Mid-level Patches in Referential Games" to appear in NeurIPS 2021

PatchGame: Learning to Signal Mid-level Patches in Referential Games This repository is the official implementation of the paper - "PatchGame: Learnin

Kamal Gupta 22 Mar 16, 2022
Benchmarking the robustness of Spatial-Temporal Models

Benchmarking the robustness of Spatial-Temporal Models This repositery contains the code for the paper Benchmarking the Robustness of Spatial-Temporal

Yi Chenyu Ian 15 Dec 16, 2022
Self-supervised Product Quantization for Deep Unsupervised Image Retrieval - ICCV2021

Self-supervised Product Quantization for Deep Unsupervised Image Retrieval Pytorch implementation of SPQ Accepted to ICCV 2021 - paper Young Kyun Jang

Young Kyun Jang 71 Dec 27, 2022
NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling @ INTERSPEECH 2021 Accepted

NU-Wave — Official PyTorch Implementation NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling Junhyeok Lee, Seungu Han @ MINDsLab Inc

MINDs Lab 242 Dec 23, 2022
CVPR2021: Temporal Context Aggregation Network for Temporal Action Proposal Refinement

Temporal Context Aggregation Network - Pytorch This repo holds the pytorch-version codes of paper: "Temporal Context Aggregation Network for Temporal

Zhiwu Qing 63 Sep 27, 2022
A Framework for Encrypted Machine Learning in TensorFlow

TF Encrypted is a framework for encrypted machine learning in TensorFlow. It looks and feels like TensorFlow, taking advantage of the ease-of-use of t

TF Encrypted 0 Jul 06, 2022
PyTorch evaluation code for Delving Deep into the Generalization of Vision Transformers under Distribution Shifts.

Out-of-distribution Generalization Investigation on Vision Transformers This repository contains PyTorch evaluation code for Delving Deep into the Gen

Chongzhi Zhang 72 Dec 13, 2022
Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations

Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations This repo contains official code for the NeurIPS 2021 paper Imi

Jiayao Zhang 2 Oct 18, 2021