Steer OpenAI's Jukebox with Music Taggers

Related tags

Deep Learningtagbox
Overview

TagBox

Steer OpenAI's Jukebox with Music Taggers!

The closest thing we have to VQGAN+CLIP for music!

Unsupervised Source Separation By Steering Pretrained Music Models

Read the paper here. Submitted to ICASSP 2022.

Abstract

We showcase an unsupervised method that repurposes deep models trained for music generation and music tagging for audio source separation, without any retraining. An audio generation model is conditioned on an input mixture, producing a latent encoding of the audio used to generate audio. This generated audio is fed to a pretrained music tagger that creates source labels. The cross-entropy loss between the tag distribution for the generated audio and a predefined distribution for an isolated source is used to guide gradient ascent in the (unchanging) latent space of the generative model. This system does not update the weights of the generative model or the tagger, and only relies on moving through the generative model's latent space to produce separated sources. We use OpenAI's Jukebox as the pretrained generative model, and we couple it with four kinds of pretrained music taggers (two architectures and two tagging datasets). Experimental results on two source separation datasets, show this approach can produce separation estimates for a wider variety of sources than any tested supervised or unsupervised system. This work points to the vast and heretofore untapped potential of large pretrained music models for audio-to-audio tasks like source separation.

Try it yourself!

Click here to see our Github repository.

Run it yourself Colab notebook here: Open in Colab

Example Output — Separation

MUSDB18 and Slakh2100 examples coming soon!

Audio examples are not displayed on https://github.com/ethman/tagbox, please click here to see the demo page.

TagBox excels in separating prominent melodies from within sparse mixtures.

Wonderwall by Oasis - Vocal Separation

Mixture


TagBox Output

hyperparam setting
fft size(s) 512, 1024, 2048
lr 10.0
steps 200
tagger model(s) fcn, hcnn, musicnn
tagger data MTAT
selected tags All vocal tags

Howl's Moving Castle, Piano & Violin Duet - Violin Separation

Mixture


TagBox Output

hyperparam setting
fft size(s) 512, 1024, 2048
lr 10.0
steps 100
tagger model(s) fcn, hcnn, musicnn
tagger data MTG-Jamendo
selected tags Violin

Smoke On The Water, by Deep Purple - Vocal Separation

Mixture


TagBox Output

hyperparam setting
fft size(s) 512, 1024, 2048
lr 5.0
steps 200
tagger model(s) fcn, hcnn
tagger data MTAT
selected tags All vocal tags

Example Output - Improving Perceptual Output & "Style Transfer"

Adding multiple FFT sizes helps with perceptual quality

Similar to multi-scale spectral losses, when we use masks with multiple FFT sizes we notice that the quality of the output increases.

Mixture


TagBox with fft_size=[1024]

Notice the warbling effects in the following example:


TagBox with fft_size=[1024, 2048]

Those warbling effects are mitigated by using two fft sizes:

These results, however, are not reflected in the SDR evaluation metrics.

"Style Transfer"

Remove the masking step enables Jukebox to generate any audio that will optimize the tag. In some situations, TagBox will pick out the melody and resynthesize it. But it adds lots of artifacts, making it sound like the audio was recorded in a snowstorm.

Mixture


"Style Transfer"

Here, we optimize the "guitar" tag without the mask. Notice that the "All it says to you" melody sounds like a guitar being plucked in a snowstorm:



Cite

If you use this your academic research, please cite the following:

@misc{manilow2021unsupervised,
  title={Unsupervised Source Separation By Steering Pretrained Music Models}, 
  author={Ethan Manilow and Patrick O'Reilly and Prem Seetharaman and Bryan Pardo},
  year={2021},
  eprint={2110.13071},
  archivePrefix={arXiv},
  primaryClass={cs.SD}
}
Owner
Ethan Manilow
PhD in the @interactiveaudiolab
Ethan Manilow
A Pytorch Implementation of Source Data-free Domain Adaptation for a Faster R-CNN

A Pytorch Implementation of Source Data-free Domain Adaptation for a Faster R-CNN Please follow Faster R-CNN and DAF to complete the environment confi

2 Jan 12, 2022
Picasso: A CUDA-based Library for Deep Learning over 3D Meshes

The Picasso Library is intended for complex real-world applications with large-scale surfaces, while it also performs impressively on the small-scale applications over synthetic shape manifolds. We h

97 Dec 01, 2022
The code for 'Deep Residual Fourier Transformation for Single Image Deblurring'

Deep Residual Fourier Transformation for Single Image Deblurring Xintian Mao, Yiming Liu, Wei Shen, Qingli Li and Yan Wang code will be released soon

145 Dec 13, 2022
FLVIS: Feedback Loop Based Visual Initial SLAM

FLVIS Feedback Loop Based Visual Inertial SLAM 1-Video EuRoC DataSet MH_05 Handheld Test in Lab FlVIS on UAV Platform 2-Relevent Publication: Under Re

UAV Lab - HKPolyU 182 Dec 04, 2022
DAFNe: A One-Stage Anchor-Free Deep Model for Oriented Object Detection

DAFNe: A One-Stage Anchor-Free Deep Model for Oriented Object Detection Code for our Paper DAFNe: A One-Stage Anchor-Free Deep Model for Oriented Obje

Steven Lang 58 Dec 19, 2022
Download from Onlyfans.com.

OnlySave: Onlyfans downloader Getting Started: Download the setup executable from the latest release. Install and run. Only works on Windows currently

4 May 30, 2022
REGTR: End-to-end Point Cloud Correspondences with Transformers

REGTR: End-to-end Point Cloud Correspondences with Transformers This repository contains the source code for REGTR. REGTR utilizes multiple transforme

Zi Jian Yew 108 Dec 17, 2022
Resources complimenting the Machine Learning Course led in the Faculty of mathematics and informatics part of Sofia University.

Machine Learning and Data Mining, Summer 2021-2022 How to learn data science and machine learning? Programming. Learn Python. Basic Statistics. Take a

Simeon Hristov 8 Oct 04, 2022
SegNet-Basic with Keras

SegNet-Basic: What is Segnet? Deep Convolutional Encoder-Decoder Architecture for Semantic Pixel-wise Image Segmentation Segnet = (Encoder + Decoder)

Yad Konrad 81 Jun 30, 2022
Dataset and Code for the paper "DepthTrack: Unveiling the Power of RGBD Tracking" (ICCV2021), and "Depth-only Object Tracking" (BMVC2021)

DeT and DOT Code and datasets for "DepthTrack: Unveiling the Power of RGBD Tracking" (ICCV2021) "Depth-only Object Tracking" (BMVC2021) @InProceedings

Yan Song 55 Dec 15, 2022
Pytorch Lightning code guideline for conferences

Deep learning project seed Use this seed to start new deep learning / ML projects. Built in setup.py Built in requirements Examples with MNIST Badges

Pytorch Lightning 1k Jan 02, 2023
Official implementation of the paper "Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering"

Light Field Networks Project Page | Paper | Data | Pretrained Models Vincent Sitzmann*, Semon Rezchikov*, William Freeman, Joshua Tenenbaum, Frédo Dur

Vincent Sitzmann 130 Dec 29, 2022
Easy and Efficient Object Detector

EOD Easy and Efficient Object Detector EOD (Easy and Efficient Object Detection) is a general object detection model production framework. It aim on p

381 Jan 01, 2023
Convnext-tf - Unofficial tensorflow keras implementation of ConvNeXt

ConvNeXt Tensorflow This is unofficial tensorflow keras implementation of ConvNe

29 Oct 06, 2022
Multi-Scale Progressive Fusion Network for Single Image Deraining

Multi-Scale Progressive Fusion Network for Single Image Deraining (MSPFN) This is an implementation of the MSPFN model proposed in the paper (Multi-Sc

Kuijiang 128 Nov 21, 2022
Finetune alexnet with tensorflow - Code for finetuning AlexNet in TensorFlow >= 1.2rc0

Finetune AlexNet with Tensorflow Update 15.06.2016 I revised the entire code base to work with the new input pipeline coming with TensorFlow = versio

Frederik Kratzert 766 Jan 04, 2023
Code for "NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video", CVPR 2021 oral

NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video Project Page | Paper NeuralRecon: Real-Time Coherent 3D Reconstruction from Mon

ZJU3DV 1.4k Dec 30, 2022
Stereo Hybrid Event-Frame (SHEF) Cameras for 3D Perception, IROS 2021

For academic use only. Stereo Hybrid Event-Frame (SHEF) Cameras for 3D Perception Ziwei Wang, Liyuan Pan, Yonhon Ng, Zheyu Zhuang and Robert Mahony Th

Ziwei Wang 11 Jan 04, 2023
Dataloader tools for language modelling

Installation: pip install lm_dataloader Design Philosophy A library to unify lm dataloading at large scale Simple interface, any tokenizer can be inte

5 Mar 25, 2022
A demo of how to use JAX to create a simple gravity simulation

JAX Gravity This repo contains a demo of how to use JAX to create a simple gravity simulation. It uses JAX's experimental ode package to solve the dif

Cristian Garcia 16 Sep 22, 2022