[CVPR 2021] MiVOS - Scribble to Mask module

Last update: Dec 22, 2022

Overview

MiVOS (CVPR 2021) - Scribble To Mask

Ho Kei Cheng, Yu-Wing Tai, Chi-Keung Tang

A simplistic network that turns scribbles to mask. It supports multi-object segmentation using soft-aggregation. Don't expect SOTA results from this model!

Overall structure and capabilities

	MiVOS	Mask-Propagation	Scribble-to-Mask
DAVIS/YouTube semi-supervised evaluation	❌	✔️	❌
DAVIS interactive evaluation	✔️	❌	❌
User interaction GUI tool	✔️	❌	❌
Dense Correspondences	❌	✔️	❌
Train propagation module	❌	✔️	❌
Train S2M (interaction) module	❌	❌	✔️
Train fusion module	✔️	❌	❌
Generate more synthetic data	✔️	❌	❌

Requirements

The package versions shown here are the ones that I used. You might not need the exact versions.

PyTorch 1.6.0
torchvision 0.7.0
opencv-contrib 4.2.0
davis-interactive (https://github.com/albertomontesg/davis-interactive)
gitpython for training
gdown for downloading pretrained models

Refer to the official PyTorch guide for installing PyTorch/torchvision. The rest can be installed by:

pip install opencv-contrib-python gitpython gdown

Pretrained model

Download and put the model in ./saves/. Alternatively use the provided download_model.py.

[OneDrive Mirror]

Interactive GUI

python interactive.py --image <image>

Controls:

Mouse Left - Draw scribbles
Mouse middle key - Switch positive/negative
Key f - Commit changes, clear scribbles
Key r - Clear everything
Key d - Switch between overlay/mask view
Key s - Save masks into a temporary output folder (./output/)

Known issues

The model almost always needs to focus on at least one object. It is very difficult to erase all existing masks from an image using scribbles.

Training

Datasets

Download and extract LVIS training set.
Download and extract a set of static image segmentation datasets. These are already downloaded for you if you used the download_datasets.py in Mask-Propagation.

├── lvis
│   ├── lvis_v1_train.json
│   └── train2017
├── Scribble-to-Mask
└── static
    ├── BIG_small
    └── ...

Commands

Use the deeplabv3plus_resnet50 pretrained model provided here.

CUDA_VISIBLE_DEVICES=0,1 OMP_NUM_THREADS=4 python -m torch.distributed.launch --master_port 9842 --nproc_per_node=2 train.py --id s2m --load_deeplab <path_to_deeplab.pth>

Credit

Deeplab implementation and pretrained model: https://github.com/VainF/DeepLabV3Plus-Pytorch.

Citation

Please cite our paper if you find this repo useful!

@inproceedings{MiVOS_2021,
  title={Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion},
  author={Cheng, Ho Kei and Tai, Yu-Wing and Tang, Chi-Keung},
  booktitle={CVPR},
  year={2021}
}

Contact: [email protected]

Comments

AttributeError: Caught AttributeError in DataLoader worker process 0

Hello! I followed the instructions of the training command, it has thrown an error about AttributeError. I put the static folder outside this repository as you mentioned. It is confusing that I can use the same datasets for the pretraining propagation module, the train.py in Mask-Propagation works fine.

opened by xwhkkk 2
git.exc.InvalidGitRepositoryError when running train.py

Hello! I followed the instruction of the training command, but it has thrown an error about GitRepositoryError. I used command : CUDA_VISIBLE_DEVICES=0,1 OMP_NUM_THREADS=4 python -m torch.distributed.launch --master_port 1842 --nproc_per_node=2 train.py --id s2m --load_deeplab ./deeplab_resnet50/best_deeplabv3plus_resnet50_voc_os16.pth, and I have 2 GPUs. Could you give me some suggestions?

opened by xwhkkk 2
About evaluation of the model

Hi,

thank you for the nice work.

I have a concern about the evaluation of the model. Because there is no validation set to pick the best model. It may has a potential overfitting problem. (Or what should the validation set for interactive segmentation look like? If there is a unified standard, it will be more helpful for everyone to compare their methods.)

In interactive object segmentation setting, is this setting popular? I am new here for the interactive segmentation. Wish to solve my concern, thank you.

opened by Limingxing00 2
Question about Local Control Strategy

A simple but practical segmentation tool! I've read your paper, and it says that local control strategy is used in S2M. However, I don't find the local control step in this code. Why don't you provide it in this tool? Will local control make significant difference to the performance?

opened by distillation-dcf 1
DeepLabv3 pre-trained models

Hello,

I wanted to mention that in order to train S2M from scratch, using the deeplabv3_resnet50 pre-trained model provided in this repo, returns the following error: KeyError: 'classifier.classifier.0.convs.0.0.weight. Meaning that the weights from this layer are not present in deeplabv3_resnet50. But using the deeplabv3plus_resnet50 from the same repo executes without errors.

Best!

opened by UndecidedBoy 1
saving error

Hello! Thanks for sharing your code. When I run python interactive.py and want to save the masks, appeared following error.

Could you give me some suggestions?

opened by xwhkkk 3
Fix simple issues and allow for cpu only use

I had to make some changes to be able to use the code on cpu only system and had troubles saving the mask from the interactive GUI and fixed it. Thanks for the great work.

opened by rami-alloush 3

Releases(1.0)

1.0(Mar 14, 2021)

Pretrained model
Source code(tar.gz)
Source code(zip)
s2m.pth(152.04 MB)

Owner

Rex Cheng

@ HKUST.

GitHub Repository https://hkchengrex.github.io/MiVOS/

Hierarchical Aggregation for 3D Instance Segmentation (ICCV 2021)

HAIS Hierarchical Aggregation for 3D Instance Segmentation (ICCV 2021) by Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, Xinggang Wang*. (*) Corresp

145 Jan 05, 2023

Brain tumor detection using Convolution-Neural Network (CNN)

Detect and Classify Brain Tumor using CNN. A system performing detection and classification by using Deep Learning Algorithms using Convolution-Neural Network (CNN).

1 Feb 07, 2022

This folder contains the implementation of the multi-relational attribute propagation algorithm.

MrAP This folder contains the implementation of the multi-relational attribute propagation algorithm. It requires the package pytorch-scatter. Please

6 Dec 06, 2022

Quantum-enhanced transformer neural network

Example of a Quantum-enhanced transformer neural network Get the code: git clone https://github.com/rdisipio/qtransformer.git cd qtransformer Create

61 Nov 08, 2022

Code for "CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds" @ICRA2021

CloudAAE This is an tensorflow implementation of "CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds" Files log:

35 Nov 14, 2022

Raptor-Multi-Tool - Raptor Multi Tool With Python

Promises 🔥 20 Stars and I'll fix every error that there is 50 Stars and we will

44 Jan 04, 2023

Implementation of Convolutional LSTM in PyTorch.

ConvLSTM_pytorch This file contains the implementation of Convolutional LSTM in PyTorch made by me and DavideA. We started from this implementation an

1.3k Dec 29, 2022

BabelCalib: A Universal Approach to Calibrating Central Cameras. In ICCV (2021)

BabelCalib: A Universal Approach to Calibrating Central Cameras This repository contains the MATLAB implementation of the BabelCalib calibration frame

55 Dec 30, 2022

Constrained Logistic Regression - How to apply specific constraints to logistic regression's coefficients

Constrained Logistic Regression Sample implementation of constructing a logistic regression with given ranges on each of the feature's coefficients (v

1 Dec 29, 2021

Convolutional Neural Network for Text Classification in Tensorflow

This code belongs to the "Implementing a CNN for Text Classification in Tensorflow" blog post. It is slightly simplified implementation of Kim's Convo

5.5k Jan 02, 2023

Includes PyTorch -> Keras model porting code for ConvNeXt family of models with fine-tuning and inference notebooks.

ConvNeXt-TF This repository provides TensorFlow / Keras implementations of different ConvNeXt [1] variants. It also provides the TensorFlow / Keras mo

87 Dec 06, 2022

This is the official pytorch implementation for the paper: Instance Similarity Learning for Unsupervised Feature Representation.

ISL This is the official pytorch implementation for the paper: Instance Similarity Learning for Unsupervised Feature Representation, which is accepted

19 May 04, 2022

[CVPR 2021] MiVOS - Scribble to Mask module

Related tags

Overview

MiVOS (CVPR 2021) - Scribble To Mask

Overall structure and capabilities

Requirements

Pretrained model

Interactive GUI

Known issues

Training

Datasets

Commands

Credit

Citation

Comments

AttributeError: Caught AttributeError in DataLoader worker process 0

git.exc.InvalidGitRepositoryError when running train.py

About evaluation of the model

Question about Local Control Strategy

DeepLabv3 pre-trained models

saving error

Fix simple issues and allow for cpu only use

Releases(1.0)

1.0(Mar 14, 2021)

Owner

Rex Cheng

Hierarchical Aggregation for 3D Instance Segmentation (ICCV 2021)

Brain tumor detection using Convolution-Neural Network (CNN)

This folder contains the implementation of the multi-relational attribute propagation algorithm.

Quantum-enhanced transformer neural network

Code for "CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds" @ICRA2021

Raptor-Multi-Tool - Raptor Multi Tool With Python

Implementation of Convolutional LSTM in PyTorch.

BabelCalib: A Universal Approach to Calibrating Central Cameras. In ICCV (2021)

Constrained Logistic Regression - How to apply specific constraints to logistic regression's coefficients

Convolutional Neural Network for Text Classification in Tensorflow

Includes PyTorch -> Keras model porting code for ConvNeXt family of models with fine-tuning and inference notebooks.

This is the official pytorch implementation for the paper: Instance Similarity Learning for Unsupervised Feature Representation.

Deep learning with dynamic computation graphs in TensorFlow

IMBENS: class-imbalanced ensemble learning in Python.

object recognition with machine learning on Respberry pi

Rule based classification A hotel s customers dataset

The implementation of ICASSP 2020 paper "Pixel-level self-paced learning for super-resolution"

End-to-End Dense Video Captioning with Parallel Decoding (ICCV 2021)

[CVPR'22] Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast

The code for our paper Semi-Supervised Learning with Multi-Head Co-Training