efficient neural audio synthesis in the waveform domain

Last update: Dec 23, 2022

Overview

neural waveshaping synthesis

real-time neural audio synthesis in the waveform domain

paper • website • colab • audio

by Ben Hayes, Charalampos Saitis, György Fazekas

This repository is the official implementation of Neural Waveshaping Synthesis.

Model Architecture

Requirements

To install:

pip install -r requirements.txt
pip install -e .

We recommend installing in a virtual environment.

Data

We trained our checkpoints on the URMP dataset. Once downloaded, the dataset can be preprocessed using scripts/create_urmp_dataset.py. This will consolidate recordings of each instrument within the dataset and preprocess them according to the pipeline in the paper.

python scripts/create_urmp_dataset.py \
  --gin-file gin/data/urmp_4second_crepe.gin \ 
  --data-directory /path/to/urmp \
  --output-directory /path/to/output \
  --device cuda:0  # torch device string for CREPE model

Alternatively, you can supply your own dataset and use the general create_dataset.py script:

python scripts/create_dataset.py \
  --gin-file gin/data/urmp_4second_crepe.gin \ 
  --data-directory /path/to/dataset \
  --output-directory /path/to/output \
  --device cuda:0  # torch device string for CREPE model

Training

To train a model on the URMP dataset, use this command:

python scripts/train.py \
  --gin-file gin/train/train_newt.gin \
  --dataset-path /path/to/processed/urmp \
  --urmp \
  --instrument vn \  # select URMP instrument with abbreviated string
  --load-data-to-memory

Or to use a non-URMP dataset:

python scripts/train.py \
  --gin-file gin/train/train_newt.gin \
  --dataset-path /path/to/processed/data \
  --load-data-to-memory

efficient neural audio synthesis in the waveform domain

Related tags

Overview

neural waveshaping synthesis

real-time neural audio synthesis in the waveform domain

paper • website • colab • audio

Model Architecture

Requirements

Data

Training

Owner

Ben Hayes

ICLR 2021, Fair Mixup: Fairness via Interpolation

Wider or Deeper: Revisiting the ResNet Model for Visual Recognition

Traffic4D: Single View Reconstruction of Repetitious Activity Using Longitudinal Self-Supervision

Identify the emotion of multiple speakers in an Audio Segment

ESTDepth: Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks (CVPR 2021)

Using pretrained language models for biomedical knowledge graph completion.

Pure python implementation reverse-mode automatic differentiation

VD-BERT: A Unified Vision and Dialog Transformer with BERT

Code to reproduce the results for Statistically Robust Neural Network Classification, published in UAI 2021

Pre-trained Deep Learning models and demos (high quality and extremely fast)

Image Matching Evaluation

[ICCV 2021 Oral] Just Ask: Learning to Answer Questions from Millions of Narrated Videos

Face Detection and Alignment using Multi-task Cascaded Convolutional Networks (MTCNN)

CZU-MHAD: A multimodal dataset for human action recognition utilizing a depth camera and 10 wearable inertial sensors

Artificial Intelligence playing minesweeper 🤖

Repositorio oficial del curso IIC2233 Programación Avanzada 🚀✨

PyDEns is a framework for solving Ordinary and Partial Differential Equations (ODEs & PDEs) using neural networks

PyVideoAI: Action Recognition Framework

Vanilla and Prototypical Networks with Random Weights for image classification on Omniglot and mini-ImageNet. Made with Python3.

A style-based Quantum Generative Adversarial Network