An executor that wraps 3D mesh models and encodes 3D content documents to d-dimension vector.

Related tags

Image Processingexecutor-3d-encoder
Overview

3D Mesh Encoder

An Executor that receives Documents containing point sets data in its blob attribute, with shape (N, 3) and encodes it to embeddings of shape (D,). Now, the following pretrained models are ready to be used to create embeddings:

  • PointConv-Shapenet-d512: A PointConv model resulted in 512 dimension of embeddings, which is finetuned based on ShapeNet dataset.
  • PointConv-Shapenet-d1024: A PointConv model resulted in 1024 dimension of embeddings, which is finetuned based on ShapeNet dataset.

Usage

via Docker image (recommended)

from jina import Flow

f = Flow().add(uses='jinahub+docker://3DMeshEncoder', \
               uses_with={'pretrained_model': 'PointConv-Shapenet-d512'})

via source code

from jina import Flow

f = Flow().add(uses='jinahub://3DMeshEncoder', \
               uses_with={'pretrained_model': 'PointConv-Shapenet-d512'})

This Executor offers a GPU tag to speed up encoding. For more information on how to run the executor on GPU, check out the documentation.

How to finetune pretrained-model?

install finetuner

$ pip install finetuner

prepare dataset

TBD...

finetuning model with labeled dataset

$ python finetune.py --help

$ python finetune.py --model_name pointconv \
    --train_dataset /path/to/train.bin \
    --eval_dataset /path/to/eval.bin \
    --batch_size 128 \
    --epochs 50

finetuning model with unlabeled dataset

$ python finetune.py --model_name pointconv \
    --train_dataset /path/to/unlabeled_data.bin \
    --interactive

References

  • PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
  • PointConv: Deep Convolutional Networks on 3D Point Clouds
Comments
  • which version should the `finetuner` module be?

    which version should the `finetuner` module be?

    I was preparing the environment of this repo. However, I found it confused that the python import code in finetune.py gives error message:

    from finetuner.tuner.callback import BestModelCheckpoint
from finetuner.tuner.pytorch.losses import TripletLoss
from finetuner.tuner.pytorch.miner import TripletEasyHardMiner

    It seems that there's no finetuner.tuner in finetuner module but finetuner.finetuner. Besides, callback/pytorch is not in finetuner.finetuner now.

    Now my finetuner module installed by running pip install -U finetuner is 0.5.0. Is there any solution? Maybe you can freeze the version in requirements.txt.

    opened by elenacliu 4
  • fix: fix previous structure

    fix: fix previous structure

    I got pretrain and finetune process mixed up, and previously I pretrained the models using finetune model structure, which might lead to worse results. The fix code is in process and may be finished in a few days.

    opened by elenacliu 2
  • Add dataset, preprocess and pointnet2

    Add dataset, preprocess and pointnet2

    Here is the modification of the structure. There are still some bugs in my pointnet2 codes. I will solve the possible problems after you comment.

    @numb3r3 @jemmyshin

    opened by elenacliu 1
  • In `farthest_point_sample` function, the first centroid sampled is not random.

    In `farthest_point_sample` function, the first centroid sampled is not random.

    I'm reading your codes.

    https://github.com/jina-ai/executor-3d-encoder/blob/main/executor/models/modules.py#L75

    The randint line is commented, but in the original implementation and the algorithm itself, it should be random? Maybe we can provide a fixed seed if we want to get the same result?

    opened by elenacliu 1
  • Provide diffusion encoder

    Provide diffusion encoder

    https://github.com/nmwsharp/diffusion-net

    I'm unfamiliar with the jina ecosystem. So this is my first issue.

    Final goal is to have dalle generate 3d digital human meshes like https://github.com/jina-ai/dalle-flow

    opened by fire 2
Releases(v0.3.0)
Owner
Jina AI
A Neural Search Company. We provide the cloud-native neural search solution powered by state-of-the-art AI technology.
Jina AI
GitHub Repository
Image Processing HighPass Filter With Python

Image_Processing_HighPassFilter High Pass Filter take the high frequency and ignore the low frequency High Pass Filter can be use to sharpening an ima

Felix Pratamasan 1 Dec 27, 2021
An application that maps an image of a LaTeX math equation to LaTeX code.

Convert images of LaTex math equations into LaTex code.

1.3k Jan 06, 2023
MikuMikuRig是一款集生成控制器，自动导入动画，自动布料为一体的blender插件

Miku_Miku_Rig MikuMikuRig是一款集生成控制器，自动导入动画，自动布料为一体的blender插件。 MikumiKurig is a Blender plugin that can generates rig, automatically imports animations

小威廉伯爵 342 Dec 29, 2022
Tools for making image cutouts from sets of TESS full frame images

Cutout tools for astronomical images Astrocut provides tools for making cutouts from sets of astronomical images with shared footprints. It is under a

Space Telescope Science Institute 20 Dec 16, 2022
Javascript image annotation tool based on image segmentation.

JS Segment Annotator Javascript image annotation tool based on image segmentation. Label image regions with mouse. Written in vanilla Javascript, with

Kota Yamaguchi 513 Nov 15, 2022
Pure Python bindings for the pure C++11/OpenCL Qrack quantum computer simulator library

pyqrack Pure Python bindings for the pure C++11/OpenCL Qrack quantum computer simulator library (PyQrack is just pure Qrack.) IMPORTANT: You must buil

vm6502q 6 Jul 21, 2022
Python Interface of P3D

pyp3d 介绍: pyp3d是一个可在python上使用的工具包，它提供了一种可使用python脚本驱动创建模型的方法，为三维建模提供了全新的思路。 pyp3d中定义了一系列建模相关的必要的数据类型，例如球体、圆锥台、四棱锥台、 拉伸体、圆角管等几何体，pyp3d还提供了许多函数来实现放置集合体、

20 Sep 07, 2022
A Python3 library to generate dynamic SVGs

The Python library for generating dynamic SVGs using Python3

1 Dec 23, 2021
Graphical tool to make photo collage posters

PhotoCollage Graphical tool to make photo collage posters PhotoCollage allows you to create photo collage posters. It assembles the input photographs

Adrien Vergé 350 Jan 02, 2023
Python Image Optimizer Script

Image-Optimizer Download and Install git clone https://github.com/stefankumpan/Image-Optimizer-Script.git cd Image-Optimizer-Script pip install -r req

Stefan Kumpan 0 Jul 15, 2021
Labelme is a graphical image annotation tool, It is written in Python and uses Qt for its graphical interface

Image Polygonal Annotation with Python (polygon, rectangle, circle, line, point and image-level flag annotation).

Kentaro Wada 9.6k Jan 09, 2023
A utility for quickly cropping large collections of images.

Crop Tool A utility for quickly cropping large collections of images. Inspired by Derrick Schultz's dataset-tools. Setup It's suggested that you use A

dusk (they/them) 6 Nov 14, 2021
thumbor is an open-source photo thumbnail service by globo.com

Survey If you use thumbor, please take 1 minute and answer this survey? It's only 2 questions and one is multiple choice!!! thumbor is a smart imaging

Thumbor (by @globocom) 9.3k Dec 31, 2022
Python script to generate vector graphics of an oriented lattice unit cell

unitcell Python script to generate vector graphics of an oriented lattice unit cell Examples unitcell --type hexagonal --eulers 12 23 34 --axes --crys

Philip Eisenlohr 2 Dec 10, 2021
👾 Python project to help you convert any image into a pixel art.

👾 Pixel Art Generator Python project to help you convert any image into a pixel art. ⚙️ Developer's Guide Things you need to get started with this co

Atul Anand 6 Dec 14, 2022
Nanosensor Image Processor (NanoImgPro), a python-based image analysis tool for dopamine nanosensors

NanoImgPro Nanosensor Image Processor (NanoImgPro), a python-based image analysis tool for dopamine nanosensors NanoImgPro.py contains the main class

1 Mar 02, 2022
A QR Code encode and decode python module

A QR Code encode and decode python module

Fayas Noushad 4 Feb 10, 2022
A warping based image translation model focusing on upper body synthesis.

Pose2Img Upper body image synthesis from skeleton(Keypoints). Sub module in the ICCV-2021 paper "Speech Drives Templates: Co-Speech Gesture Synthesis

zhiyh 15 Nov 10, 2022
Extract the temperature data of each wire from the thermal imager raw data.

Wire-Tempurature-Detection Extract the temperature data of each wire from the thermal imager raw data. The motivation of this computer vision project

JohanAckerman 1 Nov 03, 2021
An executor that wraps 3D mesh models and encodes 3D content documents to d-dimension vector.

3D Mesh Encoder An Executor that receives Documents containing point sets data in its blob attribute, with shape (N, 3) and encodes it to embeddings o

Jina AI 11 Dec 14, 2022