ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure

Last update: Dec 08, 2022

Related tags

Overview

[ 👷 🏗 👷 🏗 Coming soon! Official release with improved docs. Stay tuned. 👷 🏗 👷 🏗 ]

ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure

[]

ViViT is a collection of numerical tricks to efficiently access curvature from the generalized Gauss-Newton (GGN) matrix based on its low-rank structure. Provided functionality includes computing

GGN eigenvalues
GGN eigenpairs (eigenvalues + eigenvector)
1ˢᵗ- and 2ⁿᵈ-order directional derivatives along GGN eigenvectors
Newton steps

These operations can also further approximate the GGN to reduce cost via sub-sampling, Monte-Carlo approximation, and block-diagonal approximation.

How does it work? ViViT uses and extends BackPACK for PyTorch. The described functionality is realized through a combination of existing and new BackPACK extensions and hooks into its backpropagation.

Installation

👷 🏗 👷 🏗 The PyPI release is coming soon. 👷 🏗 👷 🏗

For now, you need to install from GitHub via

pip install vivit-for-pytorch@git+https://github.com/f-dangel/vivit.git#egg=vivit-for-pytorch

Examples

👷 🏗 👷 🏗 Coming soon! 👷 🏗 👷 🏗

How to cite

If you are using ViViT, consider citing the paper

@misc{dangel2022vivit,
      title={{ViViT}: Curvature access through the generalized Gauss-Newton's low-rank structure},
      author={Felix Dangel and Lukas Tatzel and Philipp Hennig},
      year={2022},
      eprint={2106.02624},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

Comments

[ADD] Warn about instabilities if eigenvalues are small

The directional gradient computation and transformation of the Newton step from Gram space into parameter space require division by the square root of the direction's eigenvalue. This is unstable if the eigenvalue is close to zero.

opened by f-dangel 1
[ADD] Clean `DirectionalDampedNewtonComputation`
Adds directionally damped Newton step computation with cleaned up API.

Fixes a bug in the eigenvalue criterion in the tests. It always picked one more eigenvalue than specified.
opened by f-dangel 1
[DOC] Add NTK example

Adds an example inspired by the functorch tutorial on NTKs. It demonstrates how to use vivit to compute empirical NTK matrices and makes a comparison with the functorch implementation.

opened by f-dangel 1
[ADD] Simplify `DirectionalDerivatives` API
Exotic features, like using different GGNs to compute directions and directional curvatures, as well as full control of which intermediate buffers to keep, have been deprecated in favor of a simpler API.

Remove Newton step computation for now as it was internally relying on DirectionalDerivatives

Remove many utilities and associated tests from the exotic features

Forbid duplicate indices in subsampling

Always delete intermediate buffers other than the target quantities
opened by f-dangel 1
[DOC] Set up `sphinx` and RTD

This PR adds a scaffold for the doc at https://vivit.readthedocs.io/en/latest/. Code examples are integrated via sphinx-gallery (I added a preliminary logo). Pull requests are built by the CI.

To build the docs, run make docs. You need to install the dependencies first, for example using pip install -e .[docs].

opened by f-dangel 1
Calculate Parameter Space Values of GGN Eigenvectors

The docs show how to calculate the gram matrix eigenvectors and the paper articulates that to translate from 'gram space' to parameter space we just need to multiply by the 'V' matrix.

What's the easiest way of implementing this?
question

opened by lk-wq 1
Detect loss function's `reduction`, error if unsupported
For now, the library only supports reduction='mean'. We rely on the user to use this reduction and raise awareness about this point in the documentation. It would be better to automatically have the library detect the reduction and error if it is unsupported.

This can be done via a hook into BackPACK.

[ ] Implement hook that determines the loss function reduction during backpropagation

[ ] Integrate the above hook into the *Computation and raise an exception if the reduction is not supported

[ ] Remove the comments about supported reductions in the documentation

enhancement
opened by f-dangel 0

Releases(1.0.0)

1.0.0(Jun 22, 2022)

First public release. Details about future releases will be documented in the changelog.
Source code(tar.gz)
Source code(zip)

Owner

Felix Dangel

Machine Learning PhD student at the University of Tübingen and the Max Planck Institute for Intelligent Systems.

GitHub Repository https://arxiv.org/abs/2106.02624

This repository is an implementation of paper : Improving the Training of Graph Neural Networks with Consistency Regularization

CRGNN Paper ： Improving the Training of Graph Neural Networks with Consistency Regularization Environments Implementing environment: GeForce RTX™ 3090

28 Dec 09, 2022

ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure

Related tags

Overview

ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure

Installation

Examples

How to cite

Comments

[ADD] Warn about instabilities if eigenvalues are small

[ADD] Clean `DirectionalDampedNewtonComputation`

[DOC] Add NTK example

[ADD] Simplify `DirectionalDerivatives` API

[DOC] Set up `sphinx` and RTD

Calculate Parameter Space Values of GGN Eigenvectors

Detect loss function's `reduction`, error if unsupported

Releases(1.0.0)

1.0.0(Jun 22, 2022)

Owner

Felix Dangel

The source code of the ICCV2021 paper "PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering"

CARMS: Categorical-Antithetic-REINFORCE Multi-Sample Gradient Estimator

Kaggle competition: Springleaf Marketing Response

Source code for our Paper "Learning in High-Dimensional Feature Spaces Using ANOVA-Based Matrix-Vector Multiplication"

Official repository of my book: "Deep Learning with PyTorch Step-by-Step: A Beginner's Guide"

Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification

A python package for generating, analyzing and visualizing building shadows

Pretrained models for Jax/Haiku; MobileNet, ResNet, VGG, Xception.

git《FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding》(CVPR 2021) GitHub: [fig8]

Non-Homogeneous Poisson Process Intensity Modeling and Estimation using Measure Transport

The source code for 'Noisy-Labeled NER with Confidence Estimation' accepted by NAACL 2021

StyleGAN2-ADA - Official PyTorch implementation

Garbage classification using structure data.

VLG-Net: Video-Language Graph Matching Networks for Video Grounding

Official Pytorch Implementation of Length-Adaptive Transformer (ACL 2021)

A MatConvNet-based implementation of the Fully-Convolutional Networks for image segmentation

Dataset Condensation with Contrastive Signals

Implementation of the ALPHAMEPOL algorithm, presented in Unsupervised Reinforcement Learning in Multiple Environments.

A Neural Net Training Interface on TensorFlow, with focus on speed + flexibility

This repository is an implementation of paper : Improving the Training of Graph Neural Networks with Consistency Regularization