efficient neural audio synthesis in the waveform domain

Last update: Dec 23, 2022

Overview

neural waveshaping synthesis

real-time neural audio synthesis in the waveform domain

paper • website • colab • audio

by Ben Hayes, Charalampos Saitis, György Fazekas

This repository is the official implementation of Neural Waveshaping Synthesis.

Model Architecture

Requirements

To install:

pip install -r requirements.txt
pip install -e .

We recommend installing in a virtual environment.

Data

We trained our checkpoints on the URMP dataset. Once downloaded, the dataset can be preprocessed using scripts/create_urmp_dataset.py. This will consolidate recordings of each instrument within the dataset and preprocess them according to the pipeline in the paper.

python scripts/create_urmp_dataset.py \
  --gin-file gin/data/urmp_4second_crepe.gin \ 
  --data-directory /path/to/urmp \
  --output-directory /path/to/output \
  --device cuda:0  # torch device string for CREPE model

Alternatively, you can supply your own dataset and use the general create_dataset.py script:

python scripts/create_dataset.py \
  --gin-file gin/data/urmp_4second_crepe.gin \ 
  --data-directory /path/to/dataset \
  --output-directory /path/to/output \
  --device cuda:0  # torch device string for CREPE model

Training

To train a model on the URMP dataset, use this command:

python scripts/train.py \
  --gin-file gin/train/train_newt.gin \
  --dataset-path /path/to/processed/urmp \
  --urmp \
  --instrument vn \  # select URMP instrument with abbreviated string
  --load-data-to-memory

Or to use a non-URMP dataset:

python scripts/train.py \
  --gin-file gin/train/train_newt.gin \
  --dataset-path /path/to/processed/data \
  --load-data-to-memory

efficient neural audio synthesis in the waveform domain

Related tags

Overview

neural waveshaping synthesis

real-time neural audio synthesis in the waveform domain

paper • website • colab • audio

Model Architecture

Requirements

Data

Training

Owner

Ben Hayes

Official PyTorch implementation for paper Context Matters: Graph-based Self-supervised Representation Learning for Medical Images

Open-L2O: A Comprehensive and Reproducible Benchmark for Learning to Optimize Algorithms

Weakly Supervised Learning of Rigid 3D Scene Flow

A program that uses computer vision to detect hand gestures, used for controlling movie players.

Extracts essential Mediapipe face landmarks and arranges them in a sequenced order.

Implementation of ECCV20 paper: the devil is in classification: a simple framework for long-tail object detection and instance segmentation

A state-of-the-art semi-supervised method for image recognition

Quantized models with python

Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules

Differentiable Simulation of Soft Multi-body Systems

A python library to build Model Trees with Linear Models at the leaves.

This provides the R code and data to replicate results in "The USS Trustee’s risky strategy"

Deep learning models for change detection of remote sensing images

A curated list of awesome Model-Based RL resources

A python tutorial on bayesian modeling techniques (PyMC3)

COD-Rank-Localize-and-Segment (CVPR2021)

Learning from Synthetic Data with Fine-grained Attributes for Person Re-Identification

An educational resource to help anyone learn deep reinforcement learning.

The hippynn python package - a modular library for atomistic machine learning with pytorch.

Jax/Flax implementation of Variational-DiffWave.