Codes for NeurIPS 2021 paper "On the Equivalence between Neural Network and Support Vector Machine".

Overview

On the Equivalence between Neural Network and Support Vector Machine

Codes for NeurIPS 2021 paper "On the Equivalence between Neural Network and Support Vector Machine".

Cite our paper

Yilan Chen, Wei Huang, Lam M. Nguyen, Tsui-Wei Weng, "On the Equivalence between Neural Network and Support Vector Machine", NeurIPS 2021.

@inproceedings{chen2021equiv,
  title={On the equivalence between neural network and support vector machine},
  author={Yilan Chen and Wei Huang and Lam M. Nguyen and Tsui-Wei Weng},
  booktitle={Advances in Neural Information Processing Systems},
  year={2021}
}

Overview

In this paper, we prove the equivalence between neural network (NN) and support vector machine (SVM), specifically, the infinitely wide NN trained by soft margin loss and the standard soft margin SVM with NTK trained by subgradient descent. Our main theoretical results include establishing the equivalence between NN and a broad family of L2 regularized kernel machines (KMs) with finite-width bounds, which cannot be handled by prior work, and showing that every finite-width NN trained by such regularized loss functions is approximately a KM.

Furthermore, we demonstrate our theory can enable three practical applications, including

  • non-vacuous generalization bound of NN via the corresponding KM;
  • non-trivial robustness certificate for the infinite-width NN (while existing robustness verification methods (e.g. IBP, Fast-Lin, CROWN) would provide vacuous bounds);
  • intrinsically more robust infinite-width NNs than those from previous kernel regression.

See our paper and slides for details.

Equivalence between infinite-width NNs and a family of KMs

Code overview

  • train_sgd.py: train the NN and SVM with NTK with stochastic subgradient descent. Plot the results to verify the equivalence.

  • generalization.py: compute non-vacuous generalization bound of NN via the corresponding KM.

  • regression.py: kernel ridge regression with NTK.

  • robust_svm.py:

    • test(): evaluate the robustness of NN using IBP or SVM with our method in the paper.
    • test_regressions(): evaluate the robustness of kernel ridge regression models using our method.
    • bound_ntk():calculate the lower and upper bound for NTK of two-layer fully-connected NN.
  • ibp.py: functions to calculate IBP bounds. Specified for NTK parameterization.

  • models/model.py: codes for constructing fully-connected neural networks with NTK parameterization.

  • config/:

    • svm_sgd.yaml: configurations and hyper-parameters to train NN and SVM.
    • svm_gene.yaml: configurations and hyper-parameters to calculate generalization bound.

Required environments:

This code is tested on the below environments:

python==3.8.8
torch==1.8.1
neural-tangents==0.3.6

Other required packages can be installed using Conda as follows,

conda create -n equiv-nn-svm python=3.8
conda activate equiv-nn-svm
conda install numpy tqdm matplotlib seaborn pyyaml

For the installation of PyTorch, please reference the instructions from https://pytorch.org/get-started/locally/. For the installation and usage of neural-tangents, please reference the instructions at https://github.com/google/neural-tangents.

Experiments

Train NN and SVM to verify the equivalence

python train_sgd.py

Example of the SGD results

SGD results

Example of the GD results

GD results

Computing non-vacuous generalization bound of NN via the corresponding KM

python generalization.py

Example of the generalization bound results

Generalization bound results

Robustness verification of NN

Add your paths to your NN models in the code and separate by the width. Specify the width of the models you want to verify. Then run the test() function in robust_svm.py.

python -c "import robust_svm; robust_svm.test('nn')"

Robustness verification of SVM

Add your paths to your SVM models in the code. Then run the test() function in robust_svm.py.

python -c "import robust_svm; robust_svm.test('svm')"

robustness verification results

Train kernel ridge regression with NTK models

python regression.py

Robustness verification of kernel ridge regression models

Run test_regressions() function in robust_svm.py.

python -c "import robust_svm; robust_svm.test_regressions()"

robustness verification results

Owner
Leslie
Leslie
Fine-tuning StyleGAN2 for Cartoon Face Generation

Cartoon-StyleGAN 🙃 : Fine-tuning StyleGAN2 for Cartoon Face Generation Abstract Recent studies have shown remarkable success in the unsupervised imag

Jihye Back 520 Jan 04, 2023
A Python Library for Graph Outlier Detection (Anomaly Detection)

PyGOD is a Python library for graph outlier detection (anomaly detection). This exciting yet challenging field has many key applications, e.g., detect

PyGOD Team 757 Jan 04, 2023
Code for Transformer Hawkes Process, ICML 2020.

Transformer Hawkes Process Source code for Transformer Hawkes Process (ICML 2020). Run the code Dependencies Python 3.7. Anaconda contains all the req

Simiao Zuo 111 Dec 26, 2022
A stable algorithm for GAN training

DRAGAN (Deep Regret Analytic Generative Adversarial Networks) Link to our paper - https://arxiv.org/abs/1705.07215 Pytorch implementation (thanks!) -

195 Oct 10, 2022
Official implementation of "An Image is Worth 16x16 Words, What is a Video Worth?" (2021 paper)

An Image is Worth 16x16 Words, What is a Video Worth? paper Official PyTorch Implementation Gilad Sharir, Asaf Noy, Lihi Zelnik-Manor DAMO Academy, Al

213 Nov 12, 2022
The codebase for our paper "Generative Occupancy Fields for 3D Surface-Aware Image Synthesis" (NeurIPS 2021)

Generative Occupancy Fields for 3D Surface-Aware Image Synthesis (NeurIPS 2021) Project Page | Paper Xudong Xu, Xingang Pan, Dahua Lin and Bo Dai GOF

xuxudong 97 Nov 10, 2022
Unofficial pytorch-lightning implement of Mip-NeRF

mipnerf_pl Unofficial pytorch-lightning implement of Mip-NeRF, Here are some results generated by this repository (pre-trained models are provided bel

Jianxin Huang 159 Dec 23, 2022
Website which uses Deep Learning to generate horror stories.

Creepypasta - Text Generator Website which uses Deep Learning to generate horror stories. View Demo · View Website Repo · Report Bug · Request Feature

Dhairya Sharma 5 Oct 14, 2022
Make your AirPlay devices as TTS speakers

Apple AirPlayer Home Assistant integration component, make your AirPlay devices as TTS speakers. Before Use 2021.6.X or earlier Apple Airplayer compon

George Zhao 117 Dec 15, 2022
Qt-GUI implementation of the YOLOv5 algorithm (ver.6 and ver.5)

YOLOv5-GUI 🎉 YOLOv5算法(ver.6及ver.5)的Qt-GUI实现 🎉 Qt-GUI implementation of the YOLOv5 algorithm (ver.6 and ver.5). 基于YOLOv5的v5版本和v6版本及Javacr大佬的UI逻辑进行编写

EricFang 12 Dec 28, 2022
Benchmark datasets, data loaders, and evaluators for graph machine learning

Overview The Open Graph Benchmark (OGB) is a collection of benchmark datasets, data loaders, and evaluators for graph machine learning. Datasets cover

1.5k Jan 05, 2023
Self-Supervised Multi-Frame Monocular Scene Flow (CVPR 2021)

Self-Supervised Multi-Frame Monocular Scene Flow 3D visualization of estimated depth and scene flow (overlayed with input image) from temporally conse

Visual Inference Lab @TU Darmstadt 85 Dec 22, 2022
This is the official repository of Music Playlist Title Generation: A Machine-Translation Approach.

PlyTitle_Generation This is the official repository of Music Playlist Title Generation: A Machine-Translation Approach. The paper has been accepted by

SeungHeonDoh 6 Jan 03, 2022
Attentive Implicit Representation Networks (AIR-Nets)

Attentive Implicit Representation Networks (AIR-Nets) Preprint | Supplementary | Accepted at the International Conference on 3D Vision (3DV) teaser.mo

29 Dec 07, 2022
Official PyTorch Implementation of paper "NeLF: Neural Light-transport Field for Single Portrait View Synthesis and Relighting", EGSR 2021.

NeLF: Neural Light-transport Field for Single Portrait View Synthesis and Relighting Official PyTorch Implementation of paper "NeLF: Neural Light-tran

Ken Lin 38 Dec 26, 2022
implicit displacement field

Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields [project page][paper][cite] Geometry-Consistent Neural Shape Represe

Yifan Wang 100 Dec 19, 2022
ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs

ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs This is the code of paper ConE: Cone Embeddings for Multi-Hop Reasoning over Knowl

MIRA Lab 33 Dec 07, 2022
Deep Learning Based EDM Subgenre Classification using Mel-Spectrogram and Tempogram Features"

EDM-subgenre-classifier This repository contains the code for "Deep Learning Based EDM Subgenre Classification using Mel-Spectrogram and Tempogram Fea

11 Dec 20, 2022
Dynamica causal Bayesian optimisation

Dynamic Causal Bayesian Optimization This is a Python implementation of Dynamic Causal Bayesian Optimization as presented at NeurIPS 2021. Abstract Th

nd308 18 Nov 22, 2022
Fast Differentiable Matrix Sqrt Root

Fast Differentiable Matrix Sqrt Root Geometric Interpretation of Matrix Square Root and Inverse Square Root This repository constains the official Pyt

YueSong 42 Dec 30, 2022