This is the code for our paper DAAIN: Detection of Anomalous and AdversarialInput using Normalizing Flows

Overview

Merantix-Labs: DAAIN

This is the code for our paper DAAIN: Detection of Anomalous and Adversarial Input using Normalizing Flows which can be found at arxiv.

Assumptions

There are assumptions:

  • The training data PerturbedDataset makes some assumptions about the data:
    • the ignore_index is 255
    • num_classes = 19
    • the images are resized with size == 512

Module Overview

A selection of the files with some pointers what to find where

├── configs                                   # The yaml configs
│   ├── activation_spaces
│   │   └── esp_net_256_512.yaml
│   ├── backbone
│   │   ├── esp_dropout.yaml
│   │   └── esp_net.yaml
│   ├── dataset_paths
│   │   ├── bdd100k.yaml
│   │   └── cityscapes.yaml
│   ├── data_creation.yaml                    # Used to create the training and testing data in one go
│   ├── detection_inference.yaml              # Used for inference
│   ├── detection_training.yaml               # Used for training
│   ├── esp_dropout_training.yaml             # Used to train the MC dropout baseline
│   └── paths.yaml
├── README.md                                 # This file!
├── requirements.in                           # The requirements
├── setup.py
└── src
   └── daain
       ├── backbones                          # Definitions of the backbones, currently only a slighlty modified version
       │   │                                  # of the ESPNet was tested
       │   ├── esp_dropout_net
       │   │   ├── esp_dropout_net.py
       │   │   ├── __init__.py
       │   │   ├── lightning_module.py
       │   │   └── trainer
       │   │       ├── criteria.py
       │   │       ├── data.py
       │   │       ├── dataset_collate.py
       │   │       ├── data_statistics.py
       │   │       ├── __init__.py
       │   │       ├── iou_eval.py
       │   │       ├── README.md
       │   │       ├── trainer.py            # launch this file to train the ESPDropoutNet
       │   │       ├── transformations.py
       │   │       └── visualize_graph.py
       │   └── esp_net
       │       ├── espnet.py                 # Definition of the CustomESPNet
       │       └── layers.py
       ├── baseline
       │   ├── maximum_softmax_probability.py
       │   ├── max_logit.py
       │   └── monte_carlo_dropout.py
       ├── config_schema
       ├── constants.py                      # Some constants, the last thing to refactor...
       ├── data                              # General data classes
       │   ├── datasets
       │   │   ├── bdd100k_dataset.py
       │   │   ├── cityscapes_dataset.py
       │   │   ├── labels
       │   │   │   ├── bdd100k.py
       │   │   │   ├── cityscape.py
       │   │   └── semantic_segmentation_dataset.py
       │   ├── activations_dataset.py        # This class loads the recorded activations
       │   └── perturbed_dataset.py          # This class loads the attacked images
       ├── model
       │   ├── aggregation_mode.py           # Not interesting for inference
       │   ├── classifiers.py                # All classifiers used are defined here
       │   ├── model.py                      # Probably the most important module. Check this for an example on how
       │   │                                 # to used the detection model and how to load the parts
       │   │                                 # (normalising_flow & classifier)
       │   └── normalising_flow
       │       ├── coupling_blocks
       │       │   ├── attention_blocks
       │       │   ├── causal_coupling_bock.py  # WIP
       │       │   └── subnet_constructors.py
       │       └── lightning_module.py
       ├── scripts
       │   └── data_creation.py              # Use this file to create the training and testing data
       ├── trainer                           # Trainer of the full detection model
       │   ├── data.py                       # Loading the data...
       │   └── trainer.py
       ├── utils                             # General utils
       └── visualisations                    # Visualisation helpers

Parts

In general the model consists of two parts:

  • Normalising FLow
  • Classifier / Scoring method

Both have to be trained separately, depending on the classifier. Some are parameter free (except for the threshold).

The general idea can be summarised:

  1. Record the activations of the backbone model at specific locations during a forward pass.
  2. Transform the recorded activations using a normalising flow and map them to a standard Gaussian for each variable.
  3. Apply some simple (mostly distance based) classifier on the transformed activations to get the anomaly score.

Training & Inference Process

  1. Generate perturbed and adversarial images. We do not provide code for this step.
  2. Generate the activations using src/daain/scripts/data_creation.py
  3. Train the detection model using src/daain/trainer/trainer.py
  4. Use src/daain/model/model.py to load the trained model and use it to get the anomaly score (the probability that the input was anomalous).
Owner
Merantix
Merantix
Play the Namibian game of Owela against a terrible AI. Built using Django and htmx.

Owela Club A Django project for playing the Namibian game of Owela against a dumb AI. Built following the rules described on the Mancala World wiki pa

Adam Johnson 18 Jun 01, 2022
Détection de créneaux de vaccination disponibles pour l'outil ViteMaDose

Vite Ma Dose ! est un outil open source de CovidTracker permettant de détecter les rendez-vous disponibles dans votre département afin de vous faire v

CovidTracker 239 Dec 13, 2022
Camera Intrinsic Calibration and Hand-Eye Calibration in Pybullet

This repository is mainly for camera intrinsic calibration and hand-eye calibration. Synthetic experiments are conducted in PyBullet simulator. 1. Tes

CAI Junhao 7 Oct 03, 2022
A machine learning software for extracting information from scholarly documents

GROBID GROBID documentation Visit the GROBID documentation for more detailed information. Summary GROBID (or Grobid, but not GroBid nor GroBiD) means

Patrice Lopez 1.9k Jan 08, 2023
Create single line SVG illustrations from your pictures

Create single line SVG illustrations from your pictures

Javier Bórquez 686 Dec 26, 2022
EQFace: An implementation of EQFace: A Simple Explicit Quality Network for Face Recognition

EQFace: A Simple Explicit Quality Network for Face Recognition The first face recognition network that generates explicit face quality online.

DeepCam Shenzhen 141 Dec 31, 2022
a micro OCR network with 0.07mb params.

MicroOCR a micro OCR network with 0.07mb params. Layer (type) Output Shape Param # Conv2d-1 [-1, 64, 8,

william 29 Aug 06, 2022
Semantic-based Patch Detection for Binary Programs

PMatch Semantic-based Patch Detection for Binary Programs Requirement tensorflow-gpu 1.13.1 numpy 1.16.2 scikit-learn 0.20.3 ssdeep 3.4 Usage tar -xvz

Mr.Curiosity 3 Sep 02, 2022
LEARN OPENCV IN 3 HOURS USING PYTHON - INCLUDING EXAMPLE PROJECTS

LEARN OPENCV IN 3 HOURS USING PYTHON - INCLUDING EXAMPLE PROJECTS

Murtaza Hassan 815 Dec 29, 2022
TextBoxes++: A Single-Shot Oriented Scene Text Detector

TextBoxes++: A Single-Shot Oriented Scene Text Detector Introduction This is an application for scene text detection (TextBoxes++) and recognition (CR

Minghui Liao 930 Jan 04, 2023
An organized collection of tutorials and projects created for aspriring computer vision students.

A repository created with the purpose of teaching students in BME lab 308A- Hanoi University of Science and Technology

Givralnguyen 5 Nov 24, 2021
ARU-Net - Deep Learning Chinese Word Segment

ARU-Net: A Neural Pixel Labeler for Layout Analysis of Historical Documents Contents Introduction Installation Demo Training Introduction This is the

128 Sep 12, 2022
Contextual speed detection for python

Speed Prediction using Optical Flow and 2D CNN About the challenge: Comma.AI Speed Challenge This challenge was developed by Comma.AI to predict the s

Mahimana Bhatt 2 Dec 16, 2021
An easy to use an (hopefully useful) captcha solution for pyTelegramBotAPI

pyTelegramBotCAPTCHA An easy to use and (hopefully useful) image CAPTCHA soltion for pyTelegramBotAPI. Installation: pip install pyTelegramBotCAPTCHA

29 Dec 26, 2022
The first open-source library that detects the font of a text in a image.

Typefont Typefont is an experimental library that detects the font of a text in a image. Usage Import the main function and invoke it like in the foll

Vasile Pește 1.6k Feb 24, 2022
Satoshi is a discord bot template in python using discord.py that allow you to track some live crypto prices with your own discord bot.

Satoshi ~ DiscordCryptoBot Satoshi is a simple python discord bot using discord.py that allow you to track your favorites cryptos prices with your own

Théo 2 Sep 15, 2022
computer vision, image processing and machine learning on the web browser or node.

Image processing and Machine learning labs   computer vision, image processing and machine learning on the web browser or node note Fast Fourier Trans

ryohei tanaka 487 Nov 11, 2022
轻量级公式 OCR 小工具:一键识别各类公式图片,并转换为 LaTeX 格式

QC-Formula | 青尘公式 OCR 介绍 轻量级开源公式 OCR 小工具:一键识别公式图片,并转换为 LaTeX 格式。 支持从 电脑本地 导入公式图片;(后续版本将支持直接从网页导入图片) 公式图片支持 .png / .jpg / .bmp,大小为 4M 以内均可; 支持印刷体及手写体,前

青尘工作室 26 Jan 07, 2023
scantailor - Scan Tailor is an interactive post-processing tool for scanned pages.

Scan Tailor - scantailor.org This project is no longer maintained, and has not been maintained for a while. About Scan Tailor is an interactive post-p

1.5k Dec 28, 2022
Links to awesome OCR projects

Awesome OCR This list contains links to great software tools and libraries and literature related to Optical Character Recognition (OCR). Contribution

Konstantin Baierer 2.2k Jan 02, 2023