VOS: Learning What You Don’t Know by Virtual Outlier Synthesis

Related tags

Deep Learningvos
Overview

VOS

This is the source code accompanying the paper VOS: Learning What You Don’t Know by Virtual Outlier Synthesis by Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li

The codebase is heavily based on ProbDet and Detectron2.

Dataset Preparation

PASCAL VOC

Download the processed VOC 2007 and 2012 dataset from here.

The VOC dataset folder should have the following structure:

 └── VOC_DATASET_ROOT
     |
     ├── JPEGImages
     ├── voc0712_train_all.json
     └── val_coco_format.json

COCO

Download COCO2017 dataset from the official website.

Download the OOD dataset (json file) when the in-distribution dataset is Pascal VOC from here.

Download the OOD dataset (json file) when the in-distribution dataset is BDD-100k from here.

Put the two processed OOD json files to ./anntoations

The COCO dataset folder should have the following structure:

 └── COCO_DATASET_ROOT
     |
     ├── annotations
        ├── xxx (the original json files)
        ├── instances_val2017_ood_wrt_bdd_rm_overlap.json
        └── instances_val2017_ood_rm_overlap.json
     ├── train2017
     └── val2017

BDD-100k

Donwload the BDD-100k images from the official website.

Download the processed BDD-100k json files from here and here.

The BDD dataset folder should have the following structure:

 └── BDD_DATASET_ROOT
     |
     ├── images
     ├── val_bdd_converted.json
     └── train_bdd_converted.json

OpenImages

Download our OpenImages validation splits here. We created a tarball that contains the out-of-distribution data splits used in our paper for hyperparameter tuning. Do not modify or rename the internal folders as those paths are hard coded in the dataset reader. The OpenImages dataset is created in a similar way following this paper.

The OpenImages dataset folder should have the following structure:

 └── OEPNIMAGES_DATASET_ROOT
     |
     ├── coco_classes
     └── ood_classes_rm_overlap

Before training, modify the dataset address in the ./detection/core/datasets/setup_datasets.py according to your local dataset address.

Visualization of the OOD datasets

The OOD images with respect to different in-distribution datasets can be downloaded from ID-VOC-OOD-COCO, ID-VOC-OOD-openimages, ID-BDD-OOD-COCO, ID-BDD-OOD-openimages.

Training

Firstly, enter the detection folder by running

cd detection

Vanilla Faster-RCNN with VOC as the in-distribution dataset


python train_net.py
--num-gpus 8
--config-file VOC-Detection/faster-rcnn/vanilla.yaml 
--random-seed 0 
--resume

Vanilla Faster-RCNN with BDD as the in-distribution dataset

python train_net.py 
--num-gpus 8 
--config-file BDD-Detection/faster-rcnn/vanilla.yaml 
--random-seed 0 
--resume

VOS on ResNet

python train_net_gmm.py 
--num-gpus 8 
--config-file VOC-Detection/faster-rcnn/vos.yaml 
--random-seed 0 
--resume

VOS on RegNet

Before training using the RegNet as the backbone, download the pretrained RegNet backbone from here.

python train_net_gmm.py 
--num-gpus 8 
--config-file VOC-Detection/faster-rcnn/regnetx.yaml 
--random-seed 0 
--resume

Before training on VOS, change "VOS.STARTING_ITER" and "VOS.SAMPLE_NUMBER" in the config file to the desired numbers in paper.

Evaluation

Evaluation with the in-distribution dataset to be VOC

Firstly run on the in-distribution dataset:

python apply_net.py 
--test-dataset voc_custom_val 
--config-file VOC-Detection/faster-rcnn/vos.yaml 
--inference-config Inference/standard_nms.yaml 
--random-seed 0 
--image-corruption-level 0 
--visualize 0

Then run on the OOD dataset:

python apply_net.py
--test-dataset coco_ood_val 
--config-file VOC-Detection/faster-rcnn/vos.yaml 
--inference-config Inference/standard_nms.yaml 
--random-seed 0 
--image-corruption-level 0 
--visualize 0

Obtain the metrics using:

python voc_coco_plot.py 
--name vos 
--thres xxx 
--energy 1 
--seed 0

Here the threshold is determined according to ProbDet. It will be displayed in the screen as you finish evaluating on the in-distribution dataset.

Evaluation with the in-distribution dataset to be BDD

Firstly run on the in-distribution dataset:

python apply_net.py 
--test-dataset bdd_custom_val 
--config-file VOC-Detection/faster-rcnn/vos.yaml 
--inference-config Inference/standard_nms.yaml 
--random-seed 0 
--image-corruption-level 0 
--visualize 0

Then run on the OOD dataset:

python apply_net.py 
--test-dataset coco_ood_val_bdd 
--config-file VOC-Detection/faster-rcnn/vos.yaml 
--inference-config Inference/standard_nms.yaml 
--random-seed 0 
--image-corruption-level 0 
--visualize 0

Obtain the metrics using:

python bdd_coco_plot.py
--name vos 
--thres xxx 
--energy 1 
--seed 0

Pretrained models

The pretrained models for Pascal-VOC can be downloaded from vanilla and VOS-ResNet and VOS-RegNet.

The pretrained models for BDD-100k can be downloaded from vanilla and VOS-ResNet and VOS-RegNet.

VOS on Classification models

Train on WideResNet

cd classification/CIFAR/ & 
python train_virtual.py 
--start_epoch 40 
--sample_number 1000 
--sample_from 10000 
--select 1 
--loss_weight 0.1 

where "start_epoch" denotes the starting epoch of the uncertainty regularization branch.

"sample_number" denotes the size of the in-distribution queue.

"sample_from" and "select" are used to approximate the likelihood threshold during virtual outlier synthesis.

"loss_weight" denotes the weight of the regularization loss.

Please see Section 3 and Section 4.1 in the paper for details.

Train on DenseNet

cd classification/CIFAR/ &
python train_virtual_dense.py 
--start_epoch 40 
--sample_number 1000 
--sample_from 10000 
--select 1 
--loss_weight 0.1 

Evaluation on different classifiers

cd classification/CIFAR/ & 
python test.py 
--model_name xx 
--method_name xx 
--score energy 
--num_to_avg 10

where "model_name" denotes the model architectures. ("res" denotes the WideResNet and "dense" denotes the DenseNet.)

"method_name" denotes the checkpoint name you are loading.

Pretrained models

We provide the pretrained models using WideResNet and DenseNet with the in-distribution dataset to be CIFAR-10.

Citation

If you found any part of this code is useful in your research, please consider citing our paper:

 @article{du2022vos,
      title={VOS: Learning What You Don’t Know by Virtual Outlier Synthesis}, 
      author={Du, Xuefeng and Wang, Zhaoning and Cai, Mu and Li, Yixuan},
      journal={Proceedings of the International Conference on Learning Representations},
      year={2022}
}
Owner
CS Research Group led by Prof. Sharon Li
Compare neural networks by their feature similarity

PyTorch Model Compare A tiny package to compare two neural networks in PyTorch. There are many ways to compare two neural networks, but one robust and

Anand Krishnamoorthy 181 Jan 04, 2023
A PyTorch port of the Neural 3D Mesh Renderer

Neural 3D Mesh Renderer (CVPR 2018) This repo contains a PyTorch implementation of the paper Neural 3D Mesh Renderer by Hiroharu Kato, Yoshitaka Ushik

Daniilidis Group University of Pennsylvania 1k Jan 09, 2023
This repository contains code, network definitions and pre-trained models for working on remote sensing images using deep learning

Deep learning for Earth Observation This repository contains code, network definitions and pre-trained models for working on remote sensing images usi

Nicolas Audebert 447 Jan 05, 2023
An evaluation toolkit for voice conversion models.

Voice-conversion-evaluation An evaluation toolkit for voice conversion models. Sample test pair Generate the metadata for evaluating models. The direc

30 Aug 29, 2022
This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation

SO-Pose This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation This paper is basically an

shangbuhuan 52 Nov 25, 2022
Joint Learning of 3D Shape Retrieval and Deformation, CVPR 2021

Joint Learning of 3D Shape Retrieval and Deformation Joint Learning of 3D Shape Retrieval and Deformation Mikaela Angelina Uy, Vladimir G. Kim, Minhyu

Mikaela Uy 38 Oct 18, 2022
Twins: Revisiting the Design of Spatial Attention in Vision Transformers

Twins: Revisiting the Design of Spatial Attention in Vision Transformers Very recently, a variety of vision transformer architectures for dense predic

482 Dec 18, 2022
face_recognization (FaceNet) + TFHE (HNP) + hand_face_detection (Mediapipe)

SuperControlSystem Face_Recognization (FaceNet) 面部识别 (FaceNet) Fully Homomorphic Encryption over the Torus (HNP) 环面全同态加密 (TFHE) Hand_Face_Detection (M

liziyu0104 2 Dec 30, 2021
Voxel-based Network for Shape Completion by Leveraging Edge Generation (ICCV 2021, oral)

Voxel-based Network for Shape Completion by Leveraging Edge Generation This is the PyTorch implementation for the paper "Voxel-based Network for Shape

10 Dec 04, 2022
A style-based Quantum Generative Adversarial Network

Style-qGAN A style based Quantum Generative Adversarial Network (style-qGAN) model for Monte Carlo event generation. Tutorial We have prepared a noteb

9 Nov 24, 2022
Spatial Action Maps for Mobile Manipulation (RSS 2020)

spatial-action-maps Update: Please see our new spatial-intention-maps repository, which extends this work to multi-agent settings. It contains many ne

Jimmy Wu 27 Nov 30, 2022
The official code for paper "R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling".

R2D2 This is the official code for paper titled "R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Mode

Alipay 49 Dec 17, 2022
PoseViz – Multi-person, multi-camera 3D human pose visualization tool built using Mayavi.

PoseViz – 3D Human Pose Visualizer Multi-person, multi-camera 3D human pose visualization tool built using Mayavi. As used in MeTRAbs visualizations.

István Sárándi 79 Dec 30, 2022
Face Mesh is a face geometry solution that estimates 468 3D face landmarks in real-time even on mobile devices

Face-Mesh Face Mesh is a face geometry solution that estimates 468 3D face landmarks in real-time even on mobile devices. It employs machine learning

Farnam Javadi 9 Dec 21, 2022
This repository contains code demonstrating the methods outlined in Path Signature Area-Based Causal Discovery in Coupled Time Series presented at Causal Analysis Workshop 2021.

signed-area-causal-inference This repository contains code demonstrating the methods outlined in Path Signature Area-Based Causal Discovery in Coupled

Will Glad 1 Mar 11, 2022
CondNet: Conditional Classifier for Scene Segmentation

CondNet: Conditional Classifier for Scene Segmentation Introduction The fully convolutional network (FCN) has achieved tremendous success in dense vis

ycszen 31 Jul 22, 2022
Fader Networks: Manipulating Images by Sliding Attributes - NIPS 2017

FaderNetworks PyTorch implementation of Fader Networks (NIPS 2017). Fader Networks can generate different realistic versions of images by modifying at

Facebook Research 753 Dec 23, 2022
Classic Papers for Beginners and Impact Scope for Authors.

There have been billions of academic papers around the world. However, maybe only 0.0...01% among them are valuable or are worth reading. Since our limited life has never been forever, TopPaper provi

Qiulin Zhang 228 Dec 18, 2022
RaceBERT -- A transformer based model to predict race and ethnicty from names

RaceBERT -- A transformer based model to predict race and ethnicty from names Installation pip install racebert Using a virtual environment is highly

Prasanna Parasurama 3 Nov 02, 2022
Gym for multi-agent reinforcement learning

PettingZoo is a Python library for conducting research in multi-agent reinforcement learning, akin to a multi-agent version of Gym. Our website, with

Farama Foundation 1.6k Jan 09, 2023