Localizing-Visual-Sounds-the-Hard-Way

Code and Dataset for "Localizing Visual Sounds the Hard Way".

The repo contains code and our pre-trained model.

Environment

Python 3.6.8
Pytorch 1.3.0

Flickr-SoundNet

We provide the pretrained model here.

To test the model, testing data and ground truth should be downloaded from learning to localize sound source.

Then run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --gt_path "path to ground truth" --testset "flickr"

VGG-Sound Source

We provide the pretrained model here.

To test the model, run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --testset "vggss"

(Note, some gt bounding boxes are updated recently, all results on VGG-SS cause a 2~3% difference on IoU.)

Both test data should be placed in the following structure.

data path
│
└───frames
│   │   image001.jpg
│   │   image002.jpg
│   │
└───audio
    │   audio011.wav
    │   audio012.wav

Citation

@InProceedings{Chen21,
              title        = "Localizing Visual Sounds the Hard Way",
              author       = "Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, Andrew Zisserman",
              booktitle    = "CVPR",
              year         = "2021"}

Localizing Visual Sounds the Hard Way

Related tags

Overview

Localizing-Visual-Sounds-the-Hard-Way

Environment

Flickr-SoundNet

VGG-Sound Source

Citation

Owner

Honglie Chen

An easier way to build neural search on the cloud

PyTorch Implementation for Deep Metric Learning Pipelines

[IJCAI'21] Deep Automatic Natural Image Matting

Second-order Attention Network for Single Image Super-resolution (CVPR-2019)

Python implementation of "Elliptic Fourier Features of a Closed Contour"

Code for "LoFTR: Detector-Free Local Feature Matching with Transformers", CVPR 2021

ICON: Implicit Clothed humans Obtained from Normals

Official page of Struct-MDC (RA-L'22 with IROS'22 option); Depth completion from Visual-SLAM using point & line features

Code for "The Box Size Confidence Bias Harms Your Object Detector"

CVPR 2022 "Online Convolutional Re-parameterization"

Source code for our CVPR 2019 paper - PPGNet: Learning Point-Pair Graph for Line Segment Detection

Beancount-mercury - Beancount importer for Mercury Startup Checking

This repository contains code, network definitions and pre-trained models for working on remote sensing images using deep learning

Parris, the automated infrastructure setup tool for machine learning algorithms.

Pydantic models for pywttr and aiopywttr.

ML-Decoder: Scalable and Versatile Classification Head

Udacity Suse Cloud Native Foundations Scholarship Course Walkthrough

CVPR2021 Workshop - HDRUNet: Single Image HDR Reconstruction with Denoising and Dequantization.

TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation, CVPR2022

Official Pytorch implementation of 'RoI Tanh-polar Transformer Network for Face Parsing in the Wild.'