AudioCLIP
Extending CLIP to Image, Text and Audio
This repository contains implementation of the models described in the paper arXiv:2106.13043. This work based on our previous works:
- ESResNe(X)t-fbsp: Learning Robust Time-Frequency Transformation of Audio (2021).
- ESResNet: Environmental Sound Classification Based on Visual Domain Models (2020).
Abstract
In the past, the rapidly evolving field of sound classification greatly benefited from the application of methods from other domains. Today, we observe the trend to fuse domain-specific tasks and approaches together, which provides the community with new outstanding models.
In this work, we present an extension of the CLIP model that handles audio in addition to text and images. Our proposed model incorporates the ESResNeXt audio-model into the CLIP framework using the AudioSet dataset. Such a combination enables the proposed model to perform bimodal and unimodal classification and querying, while keeping CLIP's ability to generalize to unseen datasets in a zero-shot inference fashion.
AudioCLIP achieves new state-of-the-art results in the Environmental Sound Classification (ESC) task, out-performing other approaches by reaching accuracies of 90.07% on the UrbanSound8K and 97.15% on the ESC-50 datasets. Further it sets new baselines in the zero-shot ESC-task on the same datasets (68.78% and 69.40%, respectively).
Finally, we also assess the cross-modal querying performance of the proposed model as well as the influence of full and partial training on the results. For the sake of reproducibility, our code is published.
How to Run the Model
The required Python version is >= 3.7.
AudioCLIP
On the ESC-50 dataset
python main.py --config protocols/audioclip-esc50.json --Dataset.args.root /path/to/ESC50
On the UrbanSound8K dataset
python main.py --config protocols/audioclip-us8k.json --Dataset.args.root /path/to/UrbanSound8K
Cite Us
@misc{guzhov2021audioclip,
title={AudioCLIP: Extending CLIP to Image, Text and Audio},
author={Andrey Guzhov and Federico Raue and Jörn Hees and Andreas Dengel},
year={2021},
eprint={2106.13043},
archivePrefix={arXiv},
primaryClass={cs.SD}
}
1 Aug 19, 2021
612 Jan 4, 2023
2 Mar 4, 2022
82 Dec 26, 2022
2.3k Dec 29, 2022
23 Nov 30, 2022
39 Oct 27, 2022
4.8k Dec 30, 2022
1 Feb 09, 2022
28 Nov 21, 2022
1.3k Dec 30, 2022
5 Oct 24, 2022
44 Jan 06, 2023
1 Jan 03, 2022
4 Feb 19, 2022
219 Jan 02, 2023
16.1k Jan 08, 2023
4.6k Jan 01, 2023
12 Aug 19, 2021
17 Dec 19, 2022
4.5k Dec 30, 2022
119 Oct 08, 2022
45 Oct 19, 2022
409 Oct 28, 2022
858 Dec 27, 2022
3.4k Dec 29, 2022
111 Nov 16, 2022