AudioCLIP
Extending CLIP to Image, Text and Audio
This repository contains implementation of the models described in the paper arXiv:2106.13043. This work based on our previous works:
- ESResNe(X)t-fbsp: Learning Robust Time-Frequency Transformation of Audio (2021).
- ESResNet: Environmental Sound Classification Based on Visual Domain Models (2020).
Abstract
In the past, the rapidly evolving field of sound classification greatly benefited from the application of methods from other domains. Today, we observe the trend to fuse domain-specific tasks and approaches together, which provides the community with new outstanding models.
In this work, we present an extension of the CLIP model that handles audio in addition to text and images. Our proposed model incorporates the ESResNeXt audio-model into the CLIP framework using the AudioSet dataset. Such a combination enables the proposed model to perform bimodal and unimodal classification and querying, while keeping CLIP's ability to generalize to unseen datasets in a zero-shot inference fashion.
AudioCLIP achieves new state-of-the-art results in the Environmental Sound Classification (ESC) task, out-performing other approaches by reaching accuracies of 90.07% on the UrbanSound8K and 97.15% on the ESC-50 datasets. Further it sets new baselines in the zero-shot ESC-task on the same datasets (68.78% and 69.40%, respectively).
Finally, we also assess the cross-modal querying performance of the proposed model as well as the influence of full and partial training on the results. For the sake of reproducibility, our code is published.
How to Run the Model
The required Python version is >= 3.7.
AudioCLIP
On the ESC-50 dataset
python main.py --config protocols/audioclip-esc50.json --Dataset.args.root /path/to/ESC50
On the UrbanSound8K dataset
python main.py --config protocols/audioclip-us8k.json --Dataset.args.root /path/to/UrbanSound8K
Cite Us
@misc{guzhov2021audioclip,
title={AudioCLIP: Extending CLIP to Image, Text and Audio},
author={Andrey Guzhov and Federico Raue and Jörn Hees and Andreas Dengel},
year={2021},
eprint={2106.13043},
archivePrefix={arXiv},
primaryClass={cs.SD}
}
1 Aug 19, 2021
612 Jan 4, 2023
2 Mar 4, 2022
82 Dec 26, 2022
2.3k Dec 29, 2022
23 Nov 30, 2022
39 Oct 27, 2022
4.8k Dec 30, 2022
798 Dec 30, 2022
3 Dec 28, 2021
1 Jan 03, 2022
1k Dec 30, 2022
19 Jun 30, 2022
3 Feb 18, 2022
1.2k Jan 08, 2023
0 Aug 04, 2021
303 Dec 17, 2022
171 Nov 21, 2022
77.1k Dec 31, 2022
73 Dec 11, 2022
5.2k Dec 29, 2022
2 Apr 24, 2022
21.2k Dec 30, 2022
1 Jun 28, 2022
137 Oct 26, 2022
28 Dec 07, 2022