Personal project about genus-0 meshes, spherical harmonics and a cow

Related tags

Deep Learningmesh2sh
Overview

How to transform a cow into spherical harmonics ?

Spot the cow, from Keenan Crane's blog

Spot

Context

In the field of Deep Learning, training on images or text has made enormous progress in recent years (with a lot of data available + CNN/Transformers). The results are not yet as good for other types of signals, such as videos or 3D models. For 3D models, some recent models use a graph-based approach to deal with 3D meshes, such as Polygen. However, these networks remain difficult to train. There are plenty of alternative representations that have been used to train a Deep network on 3D models: voxels, multiview, point clouds, each having their advantages and disadvantages. In this project, I wanted to try a new one. In topology, a 3D model is nothing more than a 2D surface (possibly colored) embedded into a 3D space. If the surface is closed, we can define an interior and an exterior, but that's it. It is not like a scalar field, which is defined throughout space. Since the data is 2D, it would be useful to be able to project this 3D representation in a 2D Euclidean space, on a uniform grid, like an image, to be able to use a 2D CNN to predict our 3D models.

Deep Learning models have proven effective in learning from mel-spectrograms of audio signals, combined with convolutions. How to exploit this idea for 3D models? All periodic signals can be approximated by Fourier series. We can therefore use a Fourier series to represent any periodic function in the complex plane. In geometry, the "drawing" of this function is a closed line, so it has the topology of a circle, in 2D space. I tried to generalize this idea by using meshes with a spherical topology, which I reprojected on the sphere using a conformal (angle preserving) parametrization, then for which I calculated the harmonics thanks to a single base, that of spherical harmonics.

The origin of this project is inspired by this video by 3blue1brown.

Spherical harmonics of a 3D mesh

We only use meshes that have the topology of a sphere, i.e. they must be manifold and genus 0. The main idea is to get a spherical parametrization of the mesh, to define where are the attributes of the mesh on the sphere. Then, the spherical harmonic coefficients that best fit these attributes are calculated.

The attributes that interest us to describe the structure of the mesh are:

  • Its geometric properties. We could directly give the XYZ coordinates, but thanks to the parametrization algorithm that is used, only the density of curvature is necessary. Consequently, we also need to know the area distortion, since our parametrization is not authalic (area preserving).
  • Its colors, in RGB format. For simplicity, here I use colors by vertices, and not with a UV texture, so it loses detail.
  • The vertex density of the mesh, which allows to put more vertices in areas that originally had a lot. This density is obtained using Von Mises-Fisher kernel density estimator.

Calculates the spherical parametrization of the mesh, then displays its various attributes

First step

The spherical harmonic coefficients can be represented as images, with the coefficients corresponding to m=0 on the diagonal. The low frequencies are at the top left.

Spherical harmonics coefficients amplitude as an image for each attribute

Spherical harmonic images

Reconstruction

We can reconstruct the model from the 6 sets of coefficients, which act as 6 functions on the sphere. We first make a spherical mesh inspired by what they made in "A Curvature and Density based Generative Representation of Shapes". Some points are sampled according to the vertex density function. We then construct an isotropic mesh with respect to a given density, using Centroidal Voronoi Tesselation. The colors are interpolated at each vertex.

Then the shape is obtained by reversing our spherical parametrization. The spherical parametrization uses a mean curvature flow, which is a simple spherical parametrizations. We use the conformal variant from Can Mean-Curvature Flow Be Made Non-Singular?.

Mean curvature flow equations. See Roberta Alessandroni's Introduction to mean curvature flow for more details on the notations MCF

Reconstruction of the mesh using only spherical harmonics coefficients First step

Remarks

This project is a proof of concept. It allows to represent a model which has the topology of a sphere in spherical harmonics form. The results could be more precise, first with an authalic (area-preserving) parametrization rather than a conformal (angle-preserving) one. Also, I did not try to train a neural network using this representation, because that requires too much investment. It takes some pre-processing on common 3D datasets to keep only the watertight genus-0 meshes, and then you have to do the training, which takes time. If anyone wants to try, I'd be happy to help.

I did it out of curiosity, and to gain experience, not to have an effective result. All algorithms used were coded in python/pytorch except for some solvers from SciPy and spherical harmonics functions from shtools. It makes it easier to read, but it could be faster using other libraries.

Demo

Check the demo in Google Colab : Open In Colab

To use the functions of this project you need the dependencies below. The versions indicated are those that I have used, and are only indicative.

  • python (3.9.10)
  • pytorch (1.9.1)
  • scipy (1.7.3)
  • scikit-sparse (0.4.6)
  • pyshtools (4.9.1)

To run the demo main.ipynb, you also need :

  • jupyterlab (3.2.9)
  • trimesh (3.10.0)
  • pyvista (0.33.2)
  • pythreejs (optional, 2.3.0)

You can run these lines to install everything on Linux using conda :

conda create --name mesh2sh
conda activate mesh2sh
conda install python=3.9
conda install scipy=1.7 -c anaconda
conda install pytorch=1.9 cudatoolkit=11 -c pytorch -c conda-forge
conda install gmt intel-openmp -c conda-forge
conda install pyshtools pyvista jupyterlab -c conda-forge
conda update pyshtools -c conda-forge
pip install scikit-sparse
pip install pythreejs
pip install trimesh

Then just run the demo :

jupyter notebook main.ipynb

Contribution

To run tests, you need pytest and flake8 :

pip install pytest
pip install flake8

You can check coding style using flake8 --max-line-length=120, and run tests using python -m pytest tests/ from the root folder. Also, run the demo again to check that the results are consistent

References

Cereal box identification in store shelves using computer vision and a single train image per model.

Product Recognition on Store Shelves Description You can read the task description here. Report You can read and download our report here. Step A - Mu

Nicholas Baraghini 1 Jan 21, 2022
Viewmaker Networks: Learning Views for Unsupervised Representation Learning

Viewmaker Networks: Learning Views for Unsupervised Representation Learning Alex Tamkin, Mike Wu, and Noah Goodman Paper link: https://arxiv.org/abs/2

Alex Tamkin 31 Dec 01, 2022
Official repository for "On Generating Transferable Targeted Perturbations" (ICCV 2021)

On Generating Transferable Targeted Perturbations (ICCV'21) Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli Paper:

Muzammal Naseer 46 Nov 17, 2022
Towards Implicit Text-Guided 3D Shape Generation (CVPR2022)

Towards Implicit Text-Guided 3D Shape Generation Towards Implicit Text-Guided 3D Shape Generation (CVPR2022) Code for the paper [Towards Implicit Text

55 Dec 16, 2022
The Body Part Regression (BPR) model translates the anatomy in a radiologic volume into a machine-interpretable form.

Copyright © German Cancer Research Center (DKFZ), Division of Medical Image Computing (MIC). Please make sure that your usage of this code is in compl

MIC-DKFZ 40 Dec 18, 2022
Transfer Reinforcement Learning for Differing Action Spaces via Q-Network Representations

Transfer-Learning-in-Reinforcement-Learning Transfer Reinforcement Learning for Differing Action Spaces via Q-Network Representations Final Report Tra

Trung Hieu Tran 4 Oct 17, 2022
Official PyTorch implementation of Joint Object Detection and Multi-Object Tracking with Graph Neural Networks

This is the official PyTorch implementation of our paper: "Joint Object Detection and Multi-Object Tracking with Graph Neural Networks". Our project website and video demos are here.

Richard Wang 443 Dec 06, 2022
An official reimplementation of the method described in the INTERSPEECH 2021 paper - Speech Resynthesis from Discrete Disentangled Self-Supervised Representations.

Speech Resynthesis from Discrete Disentangled Self-Supervised Representations Implementation of the method described in the Speech Resynthesis from Di

Facebook Research 253 Jan 06, 2023
Implementation of "Unsupervised Domain Adaptive 3D Detection with Multi-Level Consistency"

Unsupervised Domain Adaptive 3D Detection with Multi-Level Consistency (ICCV2021) Paper Link: https://arxiv.org/abs/2107.11355 This implementation bui

32 Nov 17, 2022
This repo is to be freely used by ML devs to check the GAN performances without coding from scratch.

GANs for Fun Created because I can! GOAL The goal of this repo is to be freely used by ML devs to check the GAN performances without coding from scrat

Sagnik Roy 13 Jan 26, 2022
The pure and clear PyTorch Distributed Training Framework.

The pure and clear PyTorch Distributed Training Framework. Introduction Requirements and Usage Dependency Dataset Basic Usage Slurm Cluster Usage Base

WILL LEE 208 Dec 20, 2022
OpenIPDM is a MATLAB open-source platform that stands for infrastructures probabilistic deterioration model

Open-Source Toolbox for Infrastructures Probabilistic Deterioration Modelling OpenIPDM is a MATLAB open-source platform that stands for infrastructure

CIVML 0 Jan 20, 2022
Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr\"om Method (NeurIPS 2021)

Skyformer This repository is the official implementation of Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Method (NeurIPS 2021).

Qi Zeng 46 Sep 20, 2022
Code and models for ICCV2021 paper "Robust Object Detection via Instance-Level Temporal Cycle Confusion".

Robust Object Detection via Instance-Level Temporal Cycle Confusion This repo contains the implementation of the ICCV 2021 paper, Robust Object Detect

Xin Wang 69 Oct 13, 2022
PyTorch reimplementation of the paper Involution: Inverting the Inherence of Convolution for Visual Recognition [CVPR 2021].

Involution: Inverting the Inherence of Convolution for Visual Recognition Unofficial PyTorch reimplementation of the paper Involution: Inverting the I

Christoph Reich 100 Dec 01, 2022
CLADE - Efficient Semantic Image Synthesis via Class-Adaptive Normalization (TPAMI 2021)

Efficient Semantic Image Synthesis via Class-Adaptive Normalization (Accepted by TPAMI)

tzt 49 Nov 17, 2022
💡 Learnergy is a Python library for energy-based machine learning models.

Learnergy: Energy-based Machine Learners Welcome to Learnergy. Did you ever reach a bottleneck in your computational experiments? Are you tired of imp

Gustavo Rosa 57 Nov 17, 2022
Pytorch implementation of Learning Rate Dropout.

Learning-Rate-Dropout Pytorch implementation of Learning Rate Dropout. Paper Link: https://arxiv.org/pdf/1912.00144.pdf Train ResNet-34 for Cifar10: r

42 Nov 25, 2022
Background Matting: The World is Your Green Screen

Background Matting: The World is Your Green Screen By Soumyadip Sengupta, Vivek Jayaram, Brian Curless, Steve Seitz, and Ira Kemelmacher-Shlizerman Th

Soumyadip Sengupta 4.6k Jan 04, 2023
A code implementation of AC-GC: Activation Compression with Guaranteed Convergence, in NeurIPS 2021.

Code For AC-GC: Lossy Activation Compression with Guaranteed Convergence This code is intended to be used as a supplemental material for submission to

Dave Evans 2 Nov 01, 2022