Geometric Sensitivity Decomposition

Overview

Geometric Sensitivity Decomposition

License: MIT

Diagram of Contribution

  1. This repo is the official implementation of A Geometric Perspective towards Neural Calibration via Sensitivity Decomposition (tian21gsd). The pape is accpted at NeurIPS 2021. as a spotlight paper.
  2. We reimplememented Exploring Covariate and Concept Shift for Out-of-Distribution Detection (tian21explore) and include it in the code base as well. The paper is accepted at NeurIPS 2021 workshop on Distribution Shift.
  3. For a brief introduction to these two papers, please visit the project page.

Create conda environment

conda env create -f requirements.yaml
conda activate gsd

Training

  1. Dataset will be automatically downloaded in the ./datasets directory the first time.
  2. We provide support for CIFAR10 and CIFAR100. Please change name in the configuration file accordingly (default: CIFAR10).
data: 
    name: cifar10 
  1. Three sample training configuration files are provided.
    • To train a vanilla model.

      python train.py --config ./configs/train/resnet_vanilla.yaml   
      
    • To train the GSD model proposed in tian21gsd.

      python train.py --config ./configs/train/resnet_gsd.yaml   
      
    • To train the Geometric ODIN model proposed in tian21exploring.

      python train.py --config ./configs/train/resnet_geo_odin.yaml   
      

Evaluation

1, We provide support for evaluation on CIFAR10, CIFAR100, CIFAR10C, CIFAR100C and SVHN. We consider both out-of-distribution (OOD) detection and confidence calibration. Models trained on different datasets will use different evaluation datasets.

OOD detection Calibration
Training Near OOD Far OOD Special ID OOD
CIFAR10 CIFAR10C CIFAR100 SVHN CIFAR100 Splits CIFAR10 CIFAR10C
CIFAR100 CIFAR100C CIFAR10 SVHN CIFAR100 CIFAR100C
  1. The eval.py file optionally calibrates a model. It 1) evaluates calibration performance and 2) saves several scores for OOD detection evaluation later.

    • Run the following commend to evaluate on a test set.

      python eval.py --config ./configs/eval/resnet_{model}.yaml 
      
    • To specify a calibration method, select the calibration attribute out of supported ones (use 'none' to avoid calibration). Note that a vanilla model can be calibrated using three supported methods, temperature scaling, matrix scaling and dirichlet scaling. GSD and Geometric ODIN use the alpha-beta scaling.

          testing: 
              calibration: temperature # ['temperature','dirichlet','matrix','alpha-beta','none'] 
    • To select a testing dataset, modify the dataset attribute. Note that the calibration dataset (specified under data: name) can be different than the testing dataset.

          testing: 
              dataset: cifar10 # cifar10, cifar100, cifar100c, cifar10c, svhn testing dataset
  2. Calibration benchmark

    • Results will be saved under ./runs/test/{data_name}/{arch}/{calibration}/{test_dataset}_calibration.txt.
    • We use Expected Calibration Error (ECE), Negative Log Likelihood and Brier score for calibration evaluation.
    • We recommend using a 5-fold evalution for in-distribution (ID) calibration benchmark because CIFAR10/100 does not have a val/test split. Note that evalx.py does not save OOD scores.
      python evalx.py --config ./configs/train/resnet_{model}.yaml 
      
    • (Optional) To use the proposed exponential mapping (tian21gsd) for calibration, set the attribute exponential_map to 0.1.
  3. Out-of-Distribution (OOD) benchmark

    • OOD evaluation needs to run eval.py two times to extract OOD scores from both the ID and OOD datasets.
    • Results will be saved under ./runs/test/{data_name}/{arch}/{calibration}/{test_dataset}_scores.csv. For example, to evaluate OOD detection performance of a vanilla model (ID:CIFAR10 vs. OOD:CIFAR10C), you need to run eval.py twice on CIFAR10 and CIFAR10C as the testing dataset. Upon completion, you will see two files, cifar10_scores.csv and cifar10c_scores.csv in the same folder.
    • After the evaluation results are saved, to calculate OOD detection performance, run calculate_ood.py and specify the conditions of the model: training set, testing set, model name and calibration method. The flags will help the function locate csv files saved in the previous step.
      python utils/calculate_ood.py --train cifar10 --test cifar10c --model resnet_vanilla --calibration none
      
    • We use AUROC and [email protected] as evaluation metrics.

Performance

  1. confidence calibration Performance of models trained on CIFAR10
accuracy ECE Nll
CIFAR10 CIFAR10C CIFAR10 CIFAR10C CIFAR10 CIFAR10C
Vanilla 96.25 69.43 0.0151 0.1433 0.1529 1.0885
Temperature Scaling 96.02 71.54 0.0028 0.0995 0.1352 0.8699
Dirichlet Scaling 95.93 71.15 0.0049 0.1135 0.1305 0.9527
GSD (tian21gsd) 96.23 71.7 0.0057 0.0439 0.1431 0.7921
Geometric ODIN (tian21explore) 95.92 70.18 0.0016 0.0454 0.1309 0.8138
  1. Out-of-Distribution Detection Performance (AUROC) of models trained on CIFAR10
AUROC score function CIFAR100 CIFAR10C SVHN
Vanilla MSP 88.33 71.49 91.88
Energy 88.11 71.94 92.88
GSD (tian21gsd) U 92.68 77.68 99.29
Geometric ODIN (tian21explore) U 92.53 78.77 99.60

Additional Resources

  1. Pretrained models
Project page of the paper 'Analyzing Perception-Distortion Tradeoff using Enhanced Perceptual Super-resolution Network' (ECCVW 2018)

EPSR (Enhanced Perceptual Super-resolution Network) paper This repo provides the test code, pretrained models, and results on benchmark datasets of ou

Subeesh Vasu 78 Nov 19, 2022
Using deep learning to predict gene structures of the coding genes in DNA sequences of Arabidopsis thaliana

DeepGeneAnnotator: A tool to annotate the gene in the genome The master thesis of the "Using deep learning to predict gene structures of the coding ge

Ching-Tien Wang 3 Sep 09, 2022
A python module for scientific analysis of 3D objects based on VTK and Numpy

A lightweight and powerful python module for scientific analysis and visualization of 3d objects.

Marco Musy 1.5k Jan 06, 2023
Optimizing Deeper Transformers on Small Datasets

DT-Fixup Optimizing Deeper Transformers on Small Datasets Paper published in ACL 2021: arXiv Detailed instructions to replicate our results in the pap

16 Nov 14, 2022
LiDAR R-CNN: An Efficient and Universal 3D Object Detector

LiDAR R-CNN: An Efficient and Universal 3D Object Detector Introduction This is the official code of LiDAR R-CNN: An Efficient and Universal 3D Object

TuSimple 295 Jan 05, 2023
A PyTorch toolkit for 2D Human Pose Estimation.

PyTorch-Pose PyTorch-Pose is a PyTorch implementation of the general pipeline for 2D single human pose estimation. The aim is to provide the interface

Wei Yang 1.1k Dec 30, 2022
An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.

NNI Doc | 简体中文 NNI (Neural Network Intelligence) is a lightweight but powerful toolkit to help users automate Feature Engineering, Neural Architecture

Microsoft 12.4k Dec 31, 2022
The official MegEngine implementation of the ICCV 2021 paper: GyroFlow: Gyroscope-Guided Unsupervised Optical Flow Learning

[ICCV 2021] GyroFlow: Gyroscope-Guided Unsupervised Optical Flow Learning This is the official implementation of our ICCV2021 paper GyroFlow. Our pres

MEGVII Research 36 Sep 07, 2022
VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning

This is a release of our VIMPAC paper to illustrate the implementations. The pretrained checkpoints and scripts will be soon open-sourced in HuggingFace transformers.

Hao Tan 74 Dec 03, 2022
QR2Pass-project - A proof of concept for an alternative (passwordless) authentication system to a web server

QR2Pass This is a proof of concept for an alternative (passwordless) authenticat

4 Dec 09, 2022
NumQMBasic - A mini-course offered to Undergrad physics students

The best way to use this material is by forking it by click the Fork button at the top, right corner. Then you will get your own copy to play with! Th

Raghu 35 Dec 05, 2022
[NeurIPS 2020] Semi-Supervision (Unlabeled Data) & Self-Supervision Improve Class-Imbalanced / Long-Tailed Learning

Rethinking the Value of Labels for Improving Class-Imbalanced Learning This repository contains the implementation code for paper: Rethinking the Valu

Yuzhe Yang 656 Dec 28, 2022
Audio2Face - Audio To Face With Python

Audio2Face Discription We create a project that transforms audio to blendshape w

FACEGOOD 724 Dec 26, 2022
Alpha-Zero - Telegram Group Manager Bot Written In Python Using Pyrogram

✨ Alpha Zero Bot ✨ Telegram Group Manager Bot + Userbot Written In Python Using

1 Feb 17, 2022
PyGAD, a Python 3 library for building the genetic algorithm and training machine learning algorithms (Keras & PyTorch).

PyGAD: Genetic Algorithm in Python PyGAD is an open-source easy-to-use Python 3 library for building the genetic algorithm and optimizing machine lear

Ahmed Gad 1.1k Dec 26, 2022
BTC-Generator - BTC Generator With Python

Что такое BTC-Generator? Это генератор чеков всеми любимого @BTC_BANKER_BOT Для

DoomGod 3 Aug 24, 2022
Fuzzing tool (TFuzz): a fuzzing tool based on program transformation

T-Fuzz T-Fuzz consists of 2 components: Fuzzing tool (TFuzz): a fuzzing tool based on program transformation Crash Analyzer (CrashAnalyzer): a tool th

HexHive 244 Nov 09, 2022
Fast, flexible and easy to use probabilistic modelling in Python.

Please consider citing the JMLR-MLOSS Manuscript if you've used pomegranate in your academic work! pomegranate is a package for building probabilistic

Jacob Schreiber 3k Dec 29, 2022
Fastquant - Backtest and optimize your trading strategies with only 3 lines of code!

fastquant 🤓 Bringing backtesting to the mainstream fastquant allows you to easily backtest investment strategies with as few as 3 lines of python cod

Lorenzo Ampil 1k Dec 29, 2022
Prototype python implementation of the ome-ngff table spec

Prototype python implementation of the ome-ngff table spec

Kevin Yamauchi 8 Nov 20, 2022