Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models

Related tags

Deep Learningpythonmachine-learningdeep-learning
Overview

Cross-framework Python Package for Evaluation of Latent-based Generative Models

Documentation Status CircleCI codecov CodeFactor License PyPI version DOI arXiv

Latte

Latte (for LATent Tensor Evaluation) is a cross-framework Python package for evaluation of latent-based generative models. Latte supports calculation of disentanglement and controllability metrics in both PyTorch (via TorchMetrics) and TensorFlow.

Installation

For developers working on local clone, cd to the repo and replace latte with .. For example, pip install .[tests]

pip install latte-metrics           # core (numpy only)
pip install latte-metrics[pytorch]  # with torchmetrics wrapper
pip install latte-metrics[keras]    # with tensorflow wrapper
pip install latte-metrics[tests]    # for testing

Running tests locally

pip install .[tests]
pytest tests/ --cov=latte

Example

Functional API

import latte
from latte.functional.disentanglement.mutual_info import mig
import numpy as np

latte.seed(42)

z = np.random.randn(16, 8)
a = np.random.randn(16, 2)

mutual_info_gap = mig(z, a, discrete=False, reg_dim=[4, 3])

Modular API

import latte
from latte.metrics.core.disentanglement import MutualInformationGap
import numpy as np

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update_state(z, attributes)

mig_val = mig.compute()

TorchMetrics API

import latte
from latte.metrics.torch.disentanglement import MutualInformationGap
import torch

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update(z, attributes)

mig_val = mig.compute()

Keras Metric API

import latte
from latte.metrics.keras.disentanglement import MutualInformationGap
from tensorflow import keras as tfk

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update_state(z, attributes)

mig_val = mig.result()

Documentation

https://latte.readthedocs.io/en/latest

Supported metrics

🧪 Beta support | ✔️ Stable | 🔨 In Progress | 🕣 In Queue | 👀 KIV |

Metric Latte Functional Latte Modular TorchMetrics Keras Metric
Disentanglement Metrics
📝 Mutual Information Gap (MIG) 🧪 🧪 🧪 ??
📝 Dependency-blind Mutual Information Gap (DMIG) 🧪 🧪 🧪 🧪
📝 Dependency-aware Mutual Information Gap (XMIG) 🧪 🧪 🧪 🧪
📝 Dependency-aware Latent Information Gap (DLIG) 🧪 🧪 🧪 🧪
📝 Separate Attribute Predictability (SAP) 🧪 🧪 🧪 🧪
📝 Modularity 🧪 🧪 🧪 🧪
📝 β-VAE Score 👀 👀 👀 👀
📝 FactorVAE Score 👀 👀 👀 👀
📝 DCI Score 👀 👀 👀 👀
📝 Interventional Robustness Score (IRS) 👀 👀 👀 👀
📝 Consistency 👀 👀 👀 👀
📝 Restrictiveness 👀 👀 👀 👀
Interpolatability Metrics
📝 Smoothness 🧪 🧪 🧪 🧪
📝 Monotonicity 🧪 🧪 🧪 🧪
📝 Latent Density Ratio 🕣 🕣 🕣 🕣
📝 Linearity 👀 👀 👀 👀

Bundled metric modules

🧪 Experimental (subject to changes) | ✔️ Stable | 🔨 In Progress | 🕣 In Queue

Metric Bundle Latte Functional Latte Modular TorchMetrics Keras Metric Included
Dependency-aware Disentanglement 🧪 🧪 🧪 🧪 MIG, DMIG, XMIG, DLIG
LIAD-based Interpolatability 🧪 🧪 🧪 🧪 Smoothness, Monotonicity

Cite

For individual metrics, please cite the paper according to the link in the 📝 icon in front of each metric.

If you find our package useful please cite our repository and arXiv preprint as

@article{
  watcharasupat2021latte,
  author = {Watcharasupat, Karn N. and Lee, Junyoung and Lerch, Alexander},
  title = {{Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models}},
  eprint={2112.10638},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url = {https://github.com/karnwatcharasupat/latte}
  doi = {10.5281/zenodo.5786402}
}
Comments
  • Documentation: Metric Descriptions

    Documentation: Metric Descriptions

    Might be nice to provide a short description for each metric in addition to the paper links. The readme might get too long with it, but either some doc in the repo or maybe on a github.io page?

    type: documentation priority: high 
    opened by alexanderlerch 2
  • Add Smoothness and Monotonicity support

    Add Smoothness and Monotonicity support

    Smoothness

    • [x] Functional API
      • [x] implementation
      • [x] tests
    • [x] Base API
      • [x] implementation
      • [x] tests
    • [x] Torch API
      • [x] implementation
      • [x] tests
    • [x] Keras API
      • [x] implementation
      • [x] tests

    Monotonicity

    • [x] Functional API
      • [x] implementation
      • [x] tests
    • [x] Base API
      • [x] implementation
      • [x] tests
    • [x] Torch API
      • [x] implementation
      • [x] tests
    • [x] Keras API
      • [x] implementation
      • [x] tests
    type: enhancement 
    opened by karnwatcharasupat 0
  • Add Modularity support

    Add Modularity support

    • [x] Functional API
      • [x] implementation
      • [x] tests
    • [x] Base API
      • [x] implementation
      • [x] tests
    • [x] Torch API
      • [x] implementation
      • [x] tests
    • [x] Keras API
      • [x] implementation
      • [x] tests
    type: enhancement 
    opened by karnwatcharasupat 0
  • Add SAP support

    Add SAP support

    • [x] Functional API
      • [x] implementation
      • [x] tests
    • [x] Base API
      • [x] implementation
      • [x] tests
    • [x] Torch API
      • [x] implementation
      • [x] tests
    • [x] Keras API
      • [x] implementation
      • [x] tests
    type: enhancement 
    opened by karnwatcharasupat 0
  • Add DMIG, DLIG, XMIG support

    Add DMIG, DLIG, XMIG support

    DMIG

    • [x] Functional API
      • [x] implementation
      • [x] tests
    • [x] Base API
      • [x] implementation
      • [x] tests
    • [x] Torch API
      • [x] implementation
      • [x] tests
    • [x] Keras API
      • [x] implementation
      • [x] tests

    XMIG

    • [x] Functional API
      • [x] implementation
      • [x] tests
    • [x] Base API
      • [x] implementation
      • [x] tests
    • [x] Torch API
      • [x] implementation
      • [x] tests
    • [x] Keras API
      • [x] implementation
      • [x] tests

    DLIG

    • [ x] Functional API
      • [x] implementation
      • [x] tests
    • [x] Base API
      • [x] implementation
      • [x] tests
    • [x] Torch API
      • [x] implementation
      • [x] tests
    • [x] Keras API
      • [x] implementation
      • [x] tests
    type: enhancement 
    opened by karnwatcharasupat 0
  • Add MIG support

    Add MIG support

    • [x] Functional API
      • [x] implementation
      • [x] tests
    • [x] Base API
      • [x] implementation
      • [x] tests
    • [x] Torch API
      • [x] implementation
      • [x] tests
    • [x] Keras API
      • [x] implementation
      • [x] tests
    type: enhancement 
    opened by karnwatcharasupat 0
  • Support issue for on-the-fly computation in TF2 graph mode

    Support issue for on-the-fly computation in TF2 graph mode

    The current delegate-to-NumPy technique used in TF is only compatible with TF2 eager mode since Tensor.numpy() would not work in graph mode. As a result, graph-mode users will only be able to use Latte in the evaluation stage when the model weights are no longer changing but not on-the-fly during the training stage.

    However, certain computation steps required for some metrics (especially MI-based ones) necessarily require scikit-learn ops and there is no (maintainable) way to create consistent TF mirrors of those functions.

    One potential solution is to wrap the core functions in tf.numpy_function or tf.py_function but we will have to figure out a way to make the wrapper less painful to implement/maintain since the variable args/kwargs option currently used by the dtype converter is not allowed in these functions. A naive workaround would be to make a tf.numpy_function wrapper for every highest-possible level function with fixed args but this would be considered a last-resort solution.

    Links:

    • https://www.tensorflow.org/api_docs/python/tf/numpy_function
    • https://www.tensorflow.org/api_docs/python/tf/py_function
    type: enhancement priority: medium !! needs more brains !! 
    opened by karnwatcharasupat 3
Releases(v0.0.1-alpha5)

  • v0.0.1-alpha5(Jan 20, 2022)

    What's Changed

    • Add contributing guide by @karnwatcharasupat in https://github.com/karnwatcharasupat/latte/pull/16
    • [ADD] add example notebooks by @karnwatcharasupat in https://github.com/karnwatcharasupat/latte/pull/18

    Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-alpha3...v0.0.1-alpha5

    Source code(tar.gz)
    Source code(zip)

  • v0.0.1-alpha3(Dec 16, 2021)

  • v0.0.1-alpha2(Dec 9, 2021)

  • v0.0.1-alpha1(Dec 1, 2021)

Owner
Karn Watcharasupat
Lab Cat 🐱🌈 | Audio Signal Processing Research Student. NTU EEE Class of 2022. Georgia Tech Music Tech Visiting Researcher.
Karn Watcharasupat
GitHub Repository https://latte.readthedocs.io/en/latest
Real-time object detection on Android using the YOLO network with TensorFlow

TensorFlow YOLO object detection on Android Source project android-yolo is the first implementation of YOLO for TensorFlow on an Android device. It is

Nataniel Ruiz 624 Jan 03, 2023
Brax is a differentiable physics engine that simulates environments made up of rigid bodies, joints, and actuators

Brax is a differentiable physics engine that simulates environments made up of rigid bodies, joints, and actuators. It's also a suite of learning algorithms to train agents to operate in these enviro

Google 1.5k Jan 02, 2023
Implementation of "GNNAutoScale: Scalable and Expressive Graph Neural Networks via Historical Embeddings" in PyTorch

PyGAS: Auto-Scaling GNNs in PyG PyGAS is the practical realization of our G NN A uto S cale (GAS) framework, which scales arbitrary message-passing GN

Matthias Fey 139 Dec 25, 2022
ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees

ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees This repository is the official implementation of the empirica

Kuan-Lin (Jason) Chen 2 Oct 02, 2022
Implementation for paper MLP-Mixer: An all-MLP Architecture for Vision

MLP Mixer Implementation for paper MLP-Mixer: An all-MLP Architecture for Vision. Give us a star if you like this repo. Author: Github: bangoc123 Emai

Ngoc Nguyen Ba 86 Dec 10, 2022
PyTorch implementation of "Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning"

Transparency-by-Design networks (TbD-nets) This repository contains code for replicating the experiments and visualizations from the paper Transparenc

David Mascharka 351 Nov 18, 2022
Automated Attendance Project Using Face Recognition

dependencies for project: cmake 3.22.1 dlib 19.22.1 face-recognition 1.3.0 openc

Rohail Taha 1 Jan 09, 2022
Official implementation of ACTION-Net: Multipath Excitation for Action Recognition (CVPR'21).

ACTION-Net Official implementation of ACTION-Net: Multipath Excitation for Action Recognition (CVPR'21). Getting Started EgoGesture data folder struct

V-Sense 171 Dec 26, 2022
a Lightweight library for sequential learning agents, including reinforcement learning

SaLinA: SaLinA - A Flexible and Simple Library for Learning Sequential Agents (including Reinforcement Learning) TL;DR salina is a lightweight library

Facebook Research 405 Dec 17, 2022
StyleGAN2 - Official TensorFlow Implementation

StyleGAN2 - Official TensorFlow Implementation

NVIDIA Research Projects 10.1k Dec 28, 2022
Memoized coduals - Shows that it is possible to implement reverse mode autodiff using a variation on the dual numbers called the codual numbers

The dual numbers can do efficient autodiff! The codual numbers are a simple meth

wlad 2 Dec 19, 2022
Official Pytorch Implementation of Unsupervised Image Denoising with Frequency Domain Knowledge

Unsupervised Image Denoising with Frequency Domain Knowledge (BMVC 2021 Oral) : Official Project Page This repository provides the official PyTorch im

Donggon Jang 12 Sep 26, 2022
The official implementation of Equalization Loss v1 & v2 (CVPR 2020, 2021) based on MMDetection.

The Equalization Losses for Long-tailed Object Detection and Instance Segmentation This repo is official implementation CVPR 2021 paper: Equalization

Jingru Tan 129 Dec 16, 2022
A Comprehensive Analysis of Weakly-Supervised Semantic Segmentation in Different Image Domains (IJCV submission)

wsss-analysis The code of: A Comprehensive Analysis of Weakly-Supervised Semantic Segmentation in Different Image Domains, arXiv pre-print 2019 paper.

Lyndon Chan 48 Dec 18, 2022
Experimental solutions to selected exercises from the book [Advances in Financial Machine Learning by Marcos Lopez De Prado]

Advances in Financial Machine Learning Exercises Experimental solutions to selected exercises from the book Advances in Financial Machine Learning by

Brian 1.4k Jan 04, 2023
🌎 The Modern Declarative Data Flow Framework for the AI Empowered Generation.

🌎 JSONClasses JSONClasses is a declarative data flow pipeline and data graph framework. Official Website: https://www.jsonclasses.com Official Docume

Fillmula Inc. 53 Dec 09, 2022
Image Fusion Transformer

Image-Fusion-Transformer Platform Python 3.7 Pytorch =1.0 Training Dataset MS-COCO 2014 (T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ram

Vibashan VS 68 Dec 23, 2022
TensorFlow-based neural network library

Sonnet Documentation | Examples Sonnet is a library built on top of TensorFlow 2 designed to provide simple, composable abstractions for machine learn

DeepMind 9.5k Jan 07, 2023
FPSAutomaticAiming——基于YOLOV5的FPS类游戏自动瞄准AI

FPSAutomaticAiming——基于YOLOV5的FPS类游戏自动瞄准AI 声明: 本项目仅限于学习交流，不可用于非法用途，包括但不限于：用于游戏外挂等，使用本项目产生的任何后果与本人无关！ 简介 本项目基于yolov5,实现了一款FPS类游戏（CF、CSGO等）的自瞄AI，本项目旨在使用现

Fabian 246 Dec 28, 2022
Doge-Prediction - Coding Club prediction ig

Doge-Prediction Coding Club prediction ig Basically: Create an application that

1 Jan 10, 2022