Localizing-Visual-Sounds-the-Hard-Way

Code and Dataset for "Localizing Visual Sounds the Hard Way".

The repo contains code and our pre-trained model.

Environment

Python 3.6.8
Pytorch 1.3.0

Flickr-SoundNet

We provide the pretrained model here.

To test the model, testing data and ground truth should be downloaded from learning to localize sound source.

Then run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --gt_path "path to ground truth" --testset "flickr"

VGG-Sound Source

We provide the pretrained model here.

To test the model, run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --testset "vggss"

(Note, some gt bounding boxes are updated recently, all results on VGG-SS cause a 2~3% difference on IoU.)

Both test data should be placed in the following structure.

data path
│
└───frames
│   │   image001.jpg
│   │   image002.jpg
│   │
└───audio
    │   audio011.wav
    │   audio012.wav

Citation

@InProceedings{Chen21,
              title        = "Localizing Visual Sounds the Hard Way",
              author       = "Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, Andrew Zisserman",
              booktitle    = "CVPR",
              year         = "2021"}

Localizing Visual Sounds the Hard Way

Related tags

Overview

Localizing-Visual-Sounds-the-Hard-Way

Environment

Flickr-SoundNet

VGG-Sound Source

Citation

Owner

Honglie Chen

Build Graph Nets in Tensorflow

HeatNet is a python package that provides tools to build, train and evaluate neural networks designed to predict extreme heat wave events globally on daily to subseasonal timescales.

Drone-based Joint Density Map Estimation, Localization and Tracking with Space-Time Multi-Scale Attention Network

The code of paper "Block Modeling-Guided Graph Convolutional Neural Networks".

We provided a matlab implementation for an evolutionary multitasking AUC optimization framework (EMTAUC).

A new data augmentation method for extreme lighting conditions.

A full pipeline AutoML tool for tabular data

Facial detection, landmark tracking and expression transfer library for Windows, Linux and Mac

Label Hallucination for Few-Shot Classification

v objective diffusion inference code for PyTorch.

ViewFormer: NeRF-free Neural Rendering from Few Images Using Transformers

"Structure-Augmented Text Representation Learning for Efficient Knowledge Graph Completion"(WWW 2021)

A large dataset of 100k Google Satellite and matching Map images, resembling pix2pix's Google Maps dataset.

Not Suitable for Work (NSFW) classification using deep neural network Caffe models.

TalkNet 2: Non-Autoregressive Depth-Wise Separable Convolutional Model for Speech Synthesis with Explicit Pitch and Duration Prediction.

ICLR 2021, Fair Mixup: Fairness via Interpolation

ANEA: Distant Supervision for Low-Resource Named Entity Recognition

A python/pytorch utility library

SE3 Pose Interp - Interpolate camera pose or trajectory in SE3, pose interpolation, trajectory interpolation